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ABSTRACT 

This final report concerns a pioneering, three-year curriculum development project 
dedicated to translating recent learning theory and laboratory findings into 
operational practice in an existing educational institution. This project produced 
a highly successful and unique course that teaches two semesters of economics that 
students can master in from six weeks to one semester's time. In the course, 
students negotiate individual learning contracts with instructors, making decisions 
on what they will study, what media they will use, and how fast they expect to 
progress. Detailed performance specifications were prepared first, and then media 
were selected and materials developed that would most effect 1 vel y and effic rentl y 
meet those specifications. The report explains how the course was developed, 
Including the rationales, methods, and procedures which were empirically tested.- 
Only a few educational development projects comparable to this in scope and goals 
have been conducted. None appears to have been documented in the form of 
developmental model for use by others, as is done here# (Certain pages may not 
be re producible due to marginal legibility.) (Author/DB) 
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PREFACE 



I The project reported here has major implications 

for many of our urgent educational problems. It deals with 
/the process of how to "guarantee" learning. The process or 
/curriculum development model, employed in the project evolved 
on the hard testing ground of operational reality — where many 
fine theoretical models have met with sharp defeat. 

The model is radically different from the conventional 
, approach of designing instruction, so it is understandable 
that the "course" it produced is radically different from 
other courses. Both the model and the course represent a 
bold attempt to make a greatly needed quantum jump in educa- 
tional practice. The results of the project — supported by 
empirical data — are highly positive and very gratifying to 
those who dedicated their efforts to its success. 

. Educators who are familiar with projects of this type 
know- the high risks of failure involved. And if indeed failure 
occurs, it takes place in the showcase of day to day school 
operations and involves many students, instructors, and admini- 
strative staff. Yet the risk is directly related to the poten- 
tial gain, such as that which resulted from this project. 

This project was initiated and funded by the U.S. 

Office of Education's Division Of Comprehensive and Vocational 
Research (National Center for Educational Research and 
Development) out of a firm belief that such risks must be taken 
if there, is to be progress. Risk taking, even for the best 
reasons, is not easy for any organization, particularly a 
federal agency involved in the sensitive field of education. 
Recognition should be given to Mr. Robert E. Pruitt, Director 
of the Division of Comprehensive and Vocational Research, and 
Dr. Richard B. Otte, the Project officer for this project, for 
not only their courage but the quality of their personal involve- 
ment in the project from its start to its completion. 

The- project was conducted in cooperation with the 
U.S. Naval Academy, with Dr. Jesse Koontz the Project Officer 
for the Navy. 
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ABSTRACT 

| This final report concerns a pioneering, three-year 
curriculum development project dedicated to translating recent 
learning theory and laboratory findings into operational 
practice in an existing educational institution. This project 
produced a highly successful and unique course that teaches 
two semesters of economics that students can master in from 
six weeks to one semester's time. In the course, students 
negotiate individual learning contracts with instructors, 
making decisions on what they will study, what media they will 
use, and how fast they expect to progress. Detailed perfor- 
mance specifications were prepared first and then media were 
selected and materials developed that would most effectively 
and efficiently meet those specifications. 

The report explains how the course was developed, including 
the rationales, methods, and procedures which were empirically 
tested. Only a few educational development projects comparable 
to this in scope and goals have been conducted. None appears 
to have been documented in* the form of developmental model for 
use by others, as is done here. 
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CHAPTER I 



INTRODUCTION 

/ . ■ 

In early 1967, the Educational Technology Center of 
Ster? L ing Institute initiated in Washington, D.C., a major, 
pioneering Multi-Media Curriculum Development Project. The 
sponsors of the MMCD project were the U.S. Office of Education 
and the U.S. Navy. The purpose of the project was to produce 
the "best possible" educational course "by utilizing simultan- 
eously each of several recent advances in the state of the 
art of educational technology. The advances to be utilized 
were identified as the following: 

. Systems and operations analysis in planning 

. Identifying and stating educational and training 
objectives 

. Rationales for media employment 

- Research in techniques for the programming and 
utilization of materials in the several media 

. Empirical process of materials revision^ 

The solicitation material expressed the fact that 
primarily these advances were (a) of a theoretical nature or 
(b) based on experiments that have yet to be applied in 
operating environments. The problem, from the standpoint of 
project activities, was to translate this research into 

^U.S. Navy Purchasing office,' solicitation material 
for procurement N00600-68-B-6022 , Step 1, July 20, 1967. 

^Ibid. , 
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technology — i.e., to apply these advances in a real-life, 
operational setting under the constraints of time, money, 
manpower, and the limits of knowledge, to produce the "best 
possible " course . 

The Educational Technology Center (ETC) proposed a 

conceptual model and a detailed technical approach that sought 

to reflect the latest advances in educational technology and 

3 

educational research. The technical approach took the form 
of a detailed plan that described how the new theoretical and 
experimental advances would be applied in the project. 

The course that resulted from this project has proved 
to be very successful. The course is both validated and 
replicable. The course, which is currently in operation, 
has been described by the Educational Technology Center with 
the following paragraphs : 

A highly individualized, "multi-media" introductory 
economics course that virtually guarantees learning has 
' been developed by the Educational Technology Center of 
Sterling Institute. This course, developed under the 
joint sponsorship of the U-S. Office of Education and 
the U. S. Navy, has been operationally tested at the 
U.S. Naval Academy in Annapolis, Maryland, since 
January, 1969. While senior economists state that 
the course teaches two semesters of economics, some 
students complete the course in one third of a semester. 



3 

Educational Technology center. Development of a 
Multi-Media Course in Economics for the United States Naval 
Academy: Technical^ Proposal 1 (Washington, D.C.: Sterling 

Institute, 1967). (Hereinafter referred to as Technical 
Proposal 1 . ) 

- 2 - 
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i and all students can complete it in a single semester. 

! A variety of media* are employed, including self- 
; instructional audio packages, self-instructional 
printed packages, computer-based simulation models 
on commercial time-sharing systems, and 16 mm. film. 

In the course, students negotiate learning contracts 
with instructors, making decisions on what they will 
study, what media they will use, how fa&t they expect 
to progress, and so forth. The student can largely 
determine what enrichment activities, if any, he will 
take part in or conduct, including any he wishes to 
originate himself. Essentially, all that is required 
for a student to "pass" the course is that he be. able 
to pass a series of comprehensive criterion-referenced 
tests relating to core requirements . 

The course consists of four major concept areas, 
Basic Economics, Macro Economics, Micrb Economics, and 
Domestic and International issues. These concept areas 
consist of thirty-five topics organized into over 1Q0 
core and enrichment "segments." Each segment, averaging 
about thirty to fifty minutes of student learning time, 
is designed to produce specific, measurable learning. 

Criterion-referenced pretests and post tests are 
used for course units, and concept areas. Practice 
problems and illustrations are embedded in the learning. 
Self-tests are strategically located so students can 
evaluate their own progress. In responding to practice 
problems and other test items, the student uses a small, 
portable response device called a QRS (Quick Response 
System) . A light comes on only when a student pres'ses 
they key for* the correct ansv7er. At the same time, the 
QRS maintains a record of both incorrect and correct 
responses, in the form of a computer-processable punched 
card . 4 



4 

Paragraphs provided by the Educational Technology 
Center, Washington, D.C., January 5, 1970. 
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In the semester ending in January of 1970, ninety- 
eight percent -of the students achieved Eighty percent or 
more of the core objectives of the course, and some ninety 
percent of the students achieved a substantial number of 
enrichment objectives (i.e. , earned sufficient enrichment 
points to raise their grade level to a "B" or more). 5 
Appendix A provides a description of the components of the 
course, while Appendix B provides comments on media appli- 
cations in the course. 



5 

The Academy gave a grade of M C" if a student achieved 
all of the core objectives and a "B" or an'"A" depending on 
how many enrichment objectives he also achieved. The over- 
whelming majority of students earned B's and A's. 
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CHAPTER XX 



j . PROJECT PLANNING 

Perhaps the fundamental reason for the success of the 
Multi-Media Curriculum Development Project was the philosophy 
underlying its preliminary planning and subsequent management. 
This philosophy may be referred to as the "systems approach." 

It included the view that a primary function of project 
planning and management was to optimize among alternatives in 
most effectively and economically achieving the specified 
goals. Thus, the identification of alternatives and alloca- 
tion of resources in a way judged to be optimal were activities 
growing out of this philosophy. 

. Since the systems approach was the guiding philosophy 
in planning and managing the project, it is important that the 

t 

view of this approach, as held by the project management, be 
described before specific project planning and management 
activities are dealt with. 



- 5 - 
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I The Systems Approach 

i 

The systems approach may be defined as the applica- 
tion of scientific methods and tools to the prediction and 
comparison of the values, effectiveness, and costs of a set 
of alternative courses of action involving man-machine 
systems.^ The systems approach- seeks to take into account 
the interrelationships of all significant elements of a 
problem and of its solution — hence the term "system." 

A system must be goal-directed. In education, the goal 
is learning. It is important to be able to know if a system 
is achieving its goals. More specifically, it should be possible 
to measure (a) when and how well a specific goal is being 
met. by the system and (b) which system elements are contribu- 
ting in an acceptable way toward meeting that goal. This calls 

* * 

for precision in designing a new system or identifying the 
significant elements in an existing system. in addition, it 
also calls for a "feedback loop," i.e., a means for measuring 

effectiveness, for evaluating the results, and for modifying or 

\ 

revising the systfem, as necessary, to better meet the goal. . 



/ 



1 Donald W. Meals, "Heuristic Models for Systems 



Planning," Phi Delta KaPpan, January, 1967. 



/ 
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Note that the term "systems , " as used in "systems 

approach," is plural. This underlines a key concept of the 

systems approach, namely that every system is a subsystem of 

a larger system. Our concern may be with a University's 

School of Education as an instructional system, or we may be 

concerned - with a specific course taught by a specific 

instructor in one department of that school. In any case, it 

is important to recognize, for planning and management, that 

each of these instructional systems is a subsystem of a larger 

* 

system, and therefore must be responsive to the needs {or 
"constraints") of that larger system. An illustration of this 
relationship is given in Figure 1 . 



O 




INSTRUCTIONAL 

SYSTEMS 



Fig. 1. — Instructional Systems as Subsystems 
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] The concept illustrated in Figure 1 is intended to 

aid the planner -man agar to broaden the scope of variables he 
must consider as possibly relevant to a problem or its solu- 
tion. The objective is to guard against overlooking key 
variables, which Goodlad,. for example, believes to be a 
primary cause of the practical ineffectiveness 'of much of the 
curriculum research being reported today. ^ It must be clearly 
understood that each "system" operates within an "environment" 

that is at least partially created by the larger system of 

* 

which it is a part. 

Let us consider the position of the instructor at a 
university. An instructor cannot develop a course and present 
it at a university with complete disregard for his fellow 
faculty members/ the department, the school, and other factors. 
While he may have much freedom in creating his course, he must 
take into account the constraints imposed by the "larger 
system." If he, fails to do .this, his course may "fail." 3 

There is a relationship between the goals stated and 
the system needed to accomplish th.e goals. in a university, 
the goals of each school should be a subset of the university 

2 

John I. Goodlad, "Curriculum: State of the Field," 

Review of Educational Research , XXXIX, No. 3 (Washington, D.C.: 
American Educ at ion al Research Association, 1969), 367-375. 

3 . 

Indeed., it may be argued that the major reason for 
the failure of innovation after innovation in our school systems 
has been because some of the roost relevant variables are being 
overlooked. 

= 2 ^ 
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goals. Similarly, the goals of a department should 'be a 
subset of the goals of the school. The point is that the 
goals selected for the larger system have an impact on what 
are acceptable goals for the smaller systems (i.e., -subsystems) 
of which it is composed. In designing an instructional system, 
one of the first activities is to ascertain the general goals 
of the larger system (e.g., the economics department) to 
help ensure that goals specified for a smaller system (e.g., 
a course within the department) will be compatible. Having 
specified ''compatible" goals, a system can be designed with 
a higher probability of meeting them.^ 

In the systems approach, system elements — people, 
materials, equipment, procedures, strategies, etc. — must be 
justified or justifiable, in a measurable way, in terms of 
their contributions , quantitatively and qualitatively , to 
achieving the specified goals. The feedback-evaluation- 
revision process is intended to determine how well a particu- 
lar component is performing. If the performance is below the 
minimum acceptable level, the element must be modified or 
eliminated (perhaps to be replaced with some alternative). 

4 , 

It is not at all unusual for an organization to have 
goals that are entirely unrealistic in terms of the system 
which exists. For example, the system needed to "eradicate 
crime" in a city may have little relationship to the existing 
system. 

- 9 - 
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Applying this to the design of an instruction system 
would mean that if the student -learning goals were not 
achieved the system elements would be evaluated and modified 
jor replaced as necessary to improve overall system performance, 
lit would mean that all components of the system-— including 
teachers and administrators — would be held accountable for- 
performing certain functions in an acceptable way. 

Preliminary Planning 

The systems approach, as described above, provided 
the framework for planning and conducting the Multi-Media 
Curriculum Development Project. Within this framework, a 
conceptual model of the end-product of the project was 
prepared and a detailed plan for producing that product was 
laid out. 

Conceptual Model 

It was concluded, after reviewing the solicitation 
material provided by the government, that it would be appropriate 
to develop a multi-media/methods course which reflected the 
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model shown in Figure 2 . It was stated in the proposal 
that a course produced in conformance with this model would 
have the following characteristics: 

. Course content would support specific behavioral 
objectives. 

. Types and conditions of learning judged most efficient 
in inducing the specified behaviors would largely 
determine the media and methods to he employed. 

. Measures of attainment of behavioral objectives 
would be quantitative, and while testing situations 
might vary, all test results would be maintained in 
computer storage. 

. The computer would use test results in conjunction 
with other data to generate prescriptions of learning 
packages for students . 

. Each learning package would be empirically validated. 

. Characteristics of the individual sample of behavioral 
objectives would provide the basis of each student's 
learning package . 

. At the end of a particular package, the student would 
receive almost emmediate knowledge of results, work 
at his own pace, and have a high degree of success 
in the learning situation . 

. Much of the instructional material would be of a 
self-instructional nature. ® 



Model for a Single Course Instructional System from 
“Designing an Organic Curriculum, 11 paper prepared by R.M. 

Morgan and D.S. Bushnell (Bureau of Research, U.S. Office of 
Education, Washington, D.C., 1966). 

Educational Technology Center, Development of a Multi ~ 
Media Course in Economics for the United States Waval Academy: 
Technical Proposal 1 (Washington, D.C.: Sterling Institute , 

1967), p. III.l. 

- 11 - 
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Fig. 2 .--Model for a Single Instructional System 

Source: Model for a Single Course Instructional System from "Designing an 

Organic Curriculum, " paper prepared by R.M. Morgan and D.s. Bushnell 
(Bureau of Research, U-S. Office of Education, Washington, D.C. , 1966) . 
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Program Plan 



, The preliminary planning activities produced a complete 

I 

"program plan" for producing a course resembling the'model. 

This plan included task descriptions, staffing estimates, 
budgets, schedules, and other comparable items bearing on 
time, money, manpower, and techniques. A total of fourteen 
tasks yjere defined, to be accomplished in three phases over 
three years. The names of the tasks planned were: 

1. Prepare Course Descriptions 

2. Define Core Behavioral Objectives 

3. Sequence Core Behavioral Objectives 

4. Specify Additional Behavioral Objectives 

5. Prepare Test Items 

6. Prepare Criterion Referenced Tests 

7. Prepare Norm Referenced Tests 

8. Select Media 

9. Prepare Materials 

10. Develop Evaluation Plan 

11. Write and Debug Computer Programs 

12. Course Validation and Design Document Revision 

13. Revise Materials 

14. prepare Final Report 
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CHAPTER III 



I PREPARE COURSE DESCRIPTION 

The first formal task, based on the program plan, was 
to "Prepare a Course Description." The scope of this task was 
much larger than its title implies, since it was directed 
toward defining the general goals of the new course, the envir- 
onment within which the new course would be presented, the 
characteristics of the target population, and other major 
variables and constraints that applied to the development of 
the course. 

Define Target Population 

A primary consideration in designing any instructional 
program is who the program is for and what these students will 
be like when they start the program. The characteristics of 
the target population have implications for virtually all 
aspects of the program design and, for this reason, it is 
important to "define" the target population at an early stage 
in the design activities. 

The specific characteristics of the target group, 
including the degree of homogeneity or heterogeneity, provide 
information needed to most effectively make decisions in 
such areas as r the following: l 

- 14 - 
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1. Knowledges and skills to exclude because the target 
group possesses them 

j 2. Entry competencies to Plan for in knowledge-skill 

areas pertinent to the 'program 

3. Probable areas for prerequisite deficiencies 

4. Attitudes to contend with, positive and negative, 
including level of achievement motivation 

5. Probable differences in learning rates or amounts 
of , learning per unit of time 

6. Appropriateness of different instructional strate- 
gies, including media applications 

The importance of such information for decision- 
making was fully understood in the MMCD Project. Accordingly, 
formal steps were taken to obtain all "readily available" 
information regarding the target population for the new multi- 
media economic analysis course at the Naval Academy. 

The types of information sought may be summarized as 
follows : 1 

1. Size and location 

a) Size of target population and rate of increase 
or decrease 

b) Location of the group 

c) The instructional "unit" size (e.g., 1,000 
freshmen college students, thirty students 
per class and six classes) , if one current' 
exists 



O 
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it should be emphasized that this refers to informa- 
tion sought. This does not mean each item of desired informa- 
tion was obtained. ' Only "readily available" data were 
obtained, and these were very 
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2. Achievement 

a) Scores (mean, range, and standard deviation) 
on standardized, national achievement- tests 

b) Scores (mean, range, and standard deviation) 
on validated local achievement tests, and on 
any tests that will provide information about 
achievement levels in those areas that the 
course will deal with. 

3. Interests, aptitudes, and personality 

a) Expressed interests of the group 

b) Group interests as measured on valid tests 

c) Aptitudes as measured on valid tests . 

d) Valid data about personality characteristics 
of group members 

(1) Kate of learning 

(2) Style of learning 

( 3 ) Autonomy ne eds 

(4) Affiliation needs 

(5) Achievement motivation 

(6) Anxiety level 

(7) Cautiousness 

(8) Original thinking 

(9) Personal relations 

4. Socio-economic information 



- 16 - 
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Resources and Constraints 



Much of the effort relating to the preparation of the 
.Course Description document involved identifying possibly 
| relevant resources and constraints/ as was originally planned. 

It is of much benefit to all subsequent planning if key resources 
and constraints can be identified early in the project/ as was 
the case in the MMCD Project. 

Constraints are the limitations imposed on a project. 

* 

from any source. The most common constraints in the project 
were limitations of: 

. time 
. money 
. personnel 
. knowledge 

These constraints were of constant concern in the 
preliminary planning and subsequent -course management; they 
were identified and reflected in the original program planning 
and were continuously monitored in subsequent project activities. 

There were a number of other constraints/ however/ 
that played an important part in project activities. The 
target population represented a major constraint. The developed 
course had to be tailored to the entry characteristics of the 
students. Further/ the course had to present a process which 
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enabled the student to successfully complete the course within 

a time limit. -And the students and instructors had to sub jec- 
i 

tively find the course "acceptable." 

Another type of consti'aint that was of key importance 
was the policies of the Academy and the specific department 
for which the course was being developed. Existing policy 
had to either be observed or changed. If it was ignored, there 
was the risk of finding out too late that it should have been 
observed. In order for a developmental project to be success- 
ful , the support of the top dec is ion-male ers is usually essen- 
tial. When policy is obviously outdated, it is probably better 
to influence a change at the top rather than to risk. ignoring 
it at the operation level. 

Other constraints taken into account included the 
relationship of the proposed course to other courses, the 
amount of study time the target students may have, the 
operating structure and procedures of the organization v/ithin 
which the course was to operate, the number and characteristics 
of the teachers, and so on. (The kinds of constraints that 
will apply and their relative significance will, of course, 
vary from project to project. ) It was thought to be very 
important to hav6 a clear idea of what actually constituted a 
constraint and what did not (i.e., was something that could be 
changed) . 



O 
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Macro-Planning Chart 



A major development was the emergence of the Macro - 

Planning Chart.- During the early weeks of the project, it 

/ 

became apparent that some form of visual planning tool was 
needed so that various parts of the course could be viewed 
in perspective. It was believed that such a tool would also 
serve as an aid in analyzing time-topip relationships and 
media-usage logistics. Since, in general, (a) the fifty-minute 
class-hour was familiar to instructors and (b) few single 
learning events could be expected to exceed fifty minutes in 
the developed course, it was decided to construct a chart that 
divided all of the student time available for the course' 
into fifty-minute "segments." This chart was given the name 
"Macro-'Planning Chart." The construction of the Macro- 
Planning Chart is further explained in the paragraphs below. 

Under the contract, a student was to spend six fifty- 
minute hours a week studying "core" material and three fifty- 
minute hours a week studying "enrichment" material or 
"remedial" material, or some combination of both. Since there 
were seventeen weeks in the semester, the total number, of "core" 
hours required for the new course was 102 (i.e., six hours per 
week times seventeen weeks). Similarly, it was expected that 
fifty-one '’remedial" hours and fifty-one "enrichment" hours 
(i.e., three hours per week times seventeen weeks, for-each) 
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would be provided. Each such fifty-minute hour was termed a 
"segment." In 'the course, therefore, there were to be 102 

i 

core segments, fifty-one enrichment segments, and fifty-one 
remedial segments . 

This segmentation plan was applied to each of the 
major • concept areas (groups of instructional topics particu- 
larly related in some way). Figure 3 shows the Macro-Planning 
Chart that was produced for Concept Area I of the course. 

Note that there were four enrichment segments provided, twelve 
core segments, and five remedial segments. 

The Macro-Planning Chart prepared for a concept area 
indicated the number of fifty-minute student hours, or segments, 
that could be devoted to enrichment, core, and remedial 
instruction. The segments were looked upon as "time buckets" 
which were to hold instructional activities, primarily in the 
form of instructional topics. The names of specific instruc- 
tional topics were assigned to specific course segments. in 
Concept Area I, for example, the topic "Specialization" was 
assigned segments numbered C6 and C7. A minimum of twenty 
percent of all segments were reserved for testing and counsel- 
ling, with the percentage varying for given concept areas. 
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Fig. 3. — Macro Planning Chart for Concept Area I 
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The Macro-Planning Chart provided a rough "time limit" 
for speci fic i nstructional topics . If one topic were assigned 
two segments, this would mean that a total of 100 (fifty plus 
fifty) instruction/learning minutes were Available for that 
topic. If two instructional topics were assigned to the same 
segment, it meant that fifty minutes of instruction/learning 
time had to be divided between the two topics. The effect was 
to cause decision-making based on trade-off analyses. Questions 
were raised about the relative importance of different topics 
and what was important and not important within a given topic. 

The Macro-Planning Chart proved to be highly useful in 
connection with determining the detailed sequencing of events 
included in the course. It became the "drafting board" of the 
detailed instructional sequencing (sequencing is discussed in 
a later section of this report) . 



O 
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CHAPTER IV 



DEFINE BEHAVIORAL OBJECTIVES 

Perhaps the most critical task in the design of an 
| instructional system to determine the behavioral objee- 
I tives. During the first phase of the mmcd Project, by far 
the greatest amount of effort was devoted to establishing and 
gaining agreement on the behavioral objectives. Indeed, of 
any problems encountered during the MMCD Project, those asso- 
ciated with the performance of this task were unquestionably 
the most difficult to resolve. 

As Gagne has pointed out, nearly every writer who has 
attempted to describe the factors to be taken into account in 
designing instructional programs has paid attention to 
defining objectives.^- Gagne, in his The Conditions of 

Learning , lists "learning objectives" as having the first 

2 

priority in .educational decisions. In the project, a major 
problem was encountered in trying to gain agreement on behavioral 



Robert M. Gagne, "The Analysis of Instructional 
Objectives for the Design of Instruction," Teaching Machines 
and Programed Learning, II: Data and Directions , ed. by 

Robert G1 aser (Washington , D. C . : DAVI , KEA, 1965), pp. 21-65. 

^Robert M. Gagne, The Conditions of Learning (New 
York: Holt, Rinehart and 'Winston, Inc., 1965), p. 263. 
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objectives. Since this problem has been widespread, the 
general background to the problem is stated before the 
attempts to resolve it are discussed. 

Background 

Robert P. Mager's Preparing Objectives for Programmed 

Instruction is perhaps the most widely used reference by 

* 

persons seeking to define behavioral objectives for their 
education or training programs. ^ This text, however, applies 
only when the objectives have been established, since it 
describes only how to state the objectives. Some who have 
used this text have confused the process of stating objec-. 
tives with the process of deciding what objectives to state. 
The literature is rich with material on how to state objec- 
tives. The situation is quite different regarding how to 
establish and gain agreement on objectives, a process which 
must precede stating the objectives. 

One of the key factors contributing to an apparent 
emphasis on stating objectives versus establishing objectives 
would seem to be the setting in which instructional objec- 
tives were first employed on a meaningful scale. This envi- 
ronment was military training research, particularly on the 
training of electronic technicians. One of the first things 

^ R. F . Mager, Preparing Objec tives for Programmed 
Instruction tSan Francisco: Fearon Publishers , 1962) . 
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done in planning sucli training was to "identify" the objec- 
tives. The objectives were identified by analyzing the task 
for which the person or persons were to be trained. -This 
"task analysis" produced, in turn, a "task description" 
which stated what a person does to satisfactorily perforin the 
task. Behavioral objectives were "identified" from the 
performances specified in such task descriptions. 

. This process of establishing the instructional objec- 
tive? was appropriate in such military training situations, 
and the process works well where training for specific jobs 
is concerned. It was not until attempts were made totransfe 
this process to "education" that severe difficulties were 
encountered. While training is often thought of as directed 
toward a specific job, education tends to be viewed as a 

process which prepares a person to live in a desired society 
4 . 

of the future. It is much easier to gam agreement on what 
an "electronics technician" must be able to do than what a 
"good citizen" must be able to do. 



^ For a discussion of the philosophical basis of 
education and the implications for education, see Melvin M. 
Turnin' s article, "Education and Educators in a' Changing 
Society (Part One)" in Preparing Educators to Meet Emerging 
Weeds, ed. by Edgar L. Morphet ana David L. Jesser fNew 
YorkV Citations Press, Scholastic Magazine, Inc., 1969). 
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In the occupational training situation, the scope of 
variables to be considered may be relatively narrow, and 
philosophical variables may not enter into consideration. 

The task description, in any case, can serve as the control- 
ling document in that situation. 

In education, it is very difficult to even gain 
agreement on what the task is, let alone its description. 

While people can agree on broad generalities of what a 
"desired future" should be, it has been a major problem to 
obtain agreement on the specific operational implications of 
such a future for the present educational process. The 
method for "preparing instructional objectives," as developed 
from the research on military training, does not provide the 
mechanism for establishing "educational" behavioral objectives. 
Moreover, an examination of the general body of literature in 
the field of education fails to reveal that a satisfactory 
mechanism has been developed. for gaining interpersonal or 
inter-group agreement on specific behavioral objectives for 
"educational" curricula . 5 

5 

The terms "interpersonal" and "inter-group" refer 
to persons in different disciplines or at different schools, 
who have an opportunity to freely reject or to accept and use 
a given set of behavioral objectives. 
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The need for establishing measurable objectives for 
education has been argued for nearly fifty years, the Winnetka 

I 

Plan during the 1920's being one of the earliest attempts at 
producing such objectives as part of a complete program. 6 
Ralph W. Tyler's work at the University of Chicago in the 
1940's provided an important impetus for developing techniques 
for establishing educational behavioral objectives. 7 A notable 
outcome of this impetus was Benjamin Bloom's "taxonomy" 
published in 1956, which was one of the first attempts to 
produce a comprehensive list of educational objectives for 

g 

general use. This and the subsequent taxonomies of educa- 
tional objectives represent a major step in translating abstract 
goals of education to more specific meaning. 



6 

See, for example, C. Washborne, Mabel Vogel, and 
W.S. Gray, "A Survey of Winnetka Public Schools," Journal of 
Educational Research Supplementary Educational Monograph 
(1926) .“ 

7 See R.W. Tyler, Basic Principles of Curriculum and 
Instruction: Syllabus for Education jS 05 (Chicago t Univer- 

sity of Chicago Press 1950) } and R.W. Tyler, "The Functions 
of Measurement in Improving Instruction," Educational Measure- 
ment , ed. by E.F. Lindquist (Washington, D.C. : American 

Council on Education, 1950). 

g 

B.S. Bloom, ed.; Taxonomy of Educational Objectives 
(New York : Longmans , Greeri^ 1956) .. 



-27- 



EDUCATIONAL TECHNOLOGY CENTER 



While Bloom's taxonomy proves to be a useful guide to 
different classes of objectives, it does not provide opera- 
tional-level objectives. Moreover, the objectives it .lists 
jiave been criticized for being ambiguous and, in general, not 
^meeting the criteria for "good" behavioral objectives* as 
described by Mager and others. ® Nor does* the taxonomy face 
the issue of how to gain agreement on specific behavioral 
objectives . 

Edling's review in 1968 of the research concerning 
educational objectives suggests that few, if any, researchers 
are investigating the question of how to gain interpersonal 
or inter-group agreement on sets of behavioral objectives . 

The literature continues to imply that the basic problem 
is how to state behavioral objectives, rather than how to 
establish them and gain agreement on them. 

It is , nevertheless , an open secret among educational 
researchers that. there have been a series of failures in 
attempts to specify educational behavioral objectives that 
are acceptable to the group of educators who would be 
expected to employ the objectives. In general, the failures 

Q ^ 

See, for example, Gagne's "The Analysis of 
Instructional Objectives for the Design of Instruction," p. 40. 

Jack V*. Edling, "Educational Ob j ecti ves and 
Educational Media," Review of Educational Research, XXXVIII, 

No. 2 (1968) . “ 
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are fewer in the hard sciences where the "task" can be agreed 
upon (e.g., arithmetic/ physics)/ but are the rule in the 
social sciences and humanities (particularly when the instruc- 
tion is aimed at high school or college students)/ where each 
instructor might visualize the "task" in a different way. 

Human beings, of course/ are not prone to publish information 
about their unsuccessful activities. 

Maguire published the results of an experiment that 
concerned a critical issue in designing curricula in the 
humanities and social sciences — the differences in value 

judgments by teachers as regards specific/ randomly selected 

11 

educational objectives. While his study is a contribution/ 
it was / nevertheless/ a paper and pencil "laboratory" experi- 
ment. Nobody really had anything to lose/ unlike the actual 
curriculum design situations where a person may believe his 
profess ional reputation is at stake . 

Project Activities 

The first phase of the Multi-Media Curriculum 
Development Project concerned preparing a definition/ or set 
of specifications/ for the desired course. A major part of 
these specifications consisted of educational behavioral 

1 7 

Thomas o. Maguire/ "Value Components of Teachers' 
Judgments of Educational Objectives," A V Co sr.mun.i cation Review , 
XVI/ No. 1 (Spring/ 19G8) . 
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objectives. In the second phase of the project, material was 
prepared to meet the specifications. The materials also were 
subjected to tryout-revision cycles until they produced the 
learning called for by the behavioral objectives. Phase 
Three required fully-operational tryouts and a formal evalu- 
ation of the highly individualized new course. in this course, 
the students have a voice in what behavioral objectives they 
will pursue , and some students complete two semesters of 



economics in one-third of a semester. 

Economics instructors from a number of universities 
took part in producing the behavioral specifications for the 
course. All told, about twenty economists had a voice in the 
course design. They represented a wide range of different 
theoretical views and practical experience in economics, with 
both the Friedman school and the Keynesian school represented. 
There were differences in personalities , politics , approach 
to life, and view of what is "good" economics. It was 
reasonable to expect great differences in value judgments 
on individual educational objectives. 

Under the contract , the new course was to teach to 
the same objectives as the comparable, existing course at 
the U. S. Naval Academy, though new points of emphasis could 
be provided. Theoretically, then, the behavioral objectives 
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could be established,- at least in part, through information 
about the existing economics course- It was soon found that 
no two instructors were in fact teaching the same . introductory 
economi cs course . 

The process of establishing behavioral objectives 
began with gaining agreement on limits of student study time, 
the ins t ructional topics to be included in the course and 
the average amount of student learning time to be devoted to 
each topic, as reflected in the Macro Planning Chart. This 
proceeded without difficulty. The next step was to establish 
the. specific behavioral objectives of the topics,* that is, to 
specify the terminal behavioral objectives, intermediate 
behavioral objectives , and entry behavior requirements for 
each topic. Problems were encountered almost immediately as 
this step got underway . 

Nature of the Problem 



A major problem was the difficulty encountered by 
some individuals of thinking in terms of what the student 
would be expected to do. In such cases, the way of thinking 
tended to concern instructor objectives, media objectives, 
and content coverage* Such thinking proved to be very 
resistant to change, despite attempts by behavioral special- 
ists to reorient such thinking to “learner" objectives. 



O 
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A second problem was the belief that students learn 
what the instructor covers. The conventionally-taught course 
at the Naval Academy "covers" two semesters of economics in 
one semester. The belief was held that the students do learn 
the material presented conventionally, though there are 
individual differences in the quality of the learning. Since 
educational technology should enable the students to learn 
faster and better, it Was believed by the faculty that behav- 
ioral objectives should be prepared for all of the material 
now covered conventionally. 

A third problem concerned how to delineate a course 
that would satisfy all of the economics faculty teaching the 
course. Each instructor seeks to teach the best economics 
course he can. This results in differences in the emphasis 
given to various points, differences in actual subject matter 
covered, and differences in the sequencing of the instruction. 
The developed course, it was found, must include the normal 
substance plus the instructors' variations in subject coverage 
and points of emphasis, if it is to be considered fully 
acceptable in terms of content coverage to each instructor. 
Such an approach, it is clear, would produce a "super-size" 
course . 
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A fourth problem concerned the human desire to take 
advantage of the course development opportunity by creating 
the highest quality course possible, with each part of the 
course being "powerful" in its own right.’ while this may or 
may not contribute to an effective course, it alone can 
increase the size of the course over any comparable conven- 
tional course because it seeks to achieve the "ideal" course, 
as seen by the faculty--!. e. , all of the content the faculty 
would like to teach but do not have sufficient time to include 
in the courses they were then teaching. 

The effect of these various problems was to cause 
packages of behavioral objectives to be repeatedly viewed by 
Academy faculty as incomplete, and therefore unacceptable, in 
terms of the desired content coverage. Course size grew as 
"content gaps" were filled by preparing additional behavioral 
objectives to meet the particular demands of individual 
instructors. 

This was the nature of the overall problem of estab- 
lishing behavioral objectives as encountered very early in 
the first phase of the project. As soon as the problem became 
apparent, a series of attempts were made to resolve it. Finally, 
a technique evolved which, in fact, did resolve the problem. 
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Attempts at Resolution 



Several approaches to establishing behavioral objec- 
tives were tried which proved to be unsuccessful in this 
project. However, each unsuccessful approach provided new 
information which finally led to producing a successful, 
approach. Accordingly, the unsuccessful attempts, as well 

li 

as the successful one, are described below. 

Approach #1 

The start of the first phase of the project was delayed 
by the government some six months, causing original arrangements 
for subject matter experts {i.e., economics instructors) to be 
lost. (Since all subject matter experts were economics instruc- 
tors, they will be referred to simply as "economists' 1 in the 
subsequent material.) As a result, the project was begun with 
fewer economists on the ETC staff than had been planned. Given 
this situation , it was decided to use a larger percentage of 
the non-economist staff members in, helping to generate behav- 
ioral objectives. These staff specialists were trained 
behavioral analysts, could write feasible, measurable, and 
observable objectives, and had individually begun the study 

of economics as part of their preliminary project activities. 

Similarly, all economists were asked to follow an ETC-prescribed 

program to orient themselves toward educational technology and 

* 

writing behavioral objectives. 
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The Educational Technology Center's economists 
anii the economics faculty readily agreed on topics and general 
goals for the topics . Much time was spent discussing how 
material in a topic was to be "handled." The ETC economists 
then conveyed a description of the goals for a given topic 
to a behavioral analyst, also referring him to material in 
standard economics texts and other sources that would be 
relevant. The behavioral analyst was to analyze the relevant 
material in terms of the stated goals and then to prepare 
behavioral objectives that represented a feasible translation 
of the goals. Each analyst was expected to interact fre- 
quently with the ETC economist responsible for his assigned 
topic, as a means of ensuring that goals were being translated 
to acceptable behaviors. 

This approach did not work, despite many revisions of 
it. Economists, whether with ETC or the economics faculty, 
found difficulty agreeing among themselves on which specific 
behavioral objectives,* from among those prepared, should be 
considered "acceptable." On the other hand, the ecojiomists 
did agree that the problem resulted from non-economists 
attempting to prepare the behavioral objectives. The 
behavioral analysts, though, stated that the economists could 
not commumicate to them the behaviors they wanted, but made 
general references to content and how it should be treated. 
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As a means of attacking the problem, stops were taken 
to (a) increase the quality of communication among the 

i 

economists and between the analysts and the economists and 
(b) provide more intensive training of economists in the 
meaning and preparation of behavioral objectives. To increase 
the quality of communication, (1) extensive meetings were 
held before, during, and after behavioral objectives were 
prepared for a particular topic, (2) detailed writeups were 
prepared by. the economists for each topic (each writeup was 
supposed to contain the terminal behavioral obj ectives for 
the topic), and (3) lengthy audio-taped oral statements be- 
tween parties regarding "agreed-upon" content requirements 
and terminal behaviors were made. 

None of these, however, was effective. it seemed that 
the economists could not write behavioral objectives accept- 
able to the behavioral specialists, and the behavioral spe- 
cialists could not write behavioral objectives acceptable to 
the economists. In general, the analysts continued to talk 

"measurable behaviors," while the economists continued to talk 
"content." While there was much talk between them, there was 
little communication. 

The economists believed that the way to resolve the 
problem was for them to tutor the analysts in economics so 
that the analysts could understand what the economists were 
trying to tell them. The analysts, on the other hand, were 
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convinced that the economists still had not accepted the 
notion of behavioral objectives and that, until they did, 
there could be no real progress. The analysts stressed that, 
at minimum, the economists must prepare the specific, terminal 
performance (behavioral) objectives for each topic. 

Much effort was spent attempting t 0 teach economists 
how to write "good" behavioral objectives, jn general, the 
more teaching experience an economist had, the more difficulty 
he appeared to have in accepting or writing behavioral objec- 
tives (but there'were some very notable exceptions to this). 
This created an awkward situation in which the junior econo- 
mists, in the view of behavioral specialists, acquired an 
understanding of behavioral objectives — indeed, were able to 
write "good" objectives— long before the senior economists did. 

The initial reaction of the senior economists toward 
behavioral objectives was largely unfavorable, and this 
attitude seemed to persist throughout approach #1. During 
this period of several weeks, no senior economist was able to 
write objectives that, according to the behavioral specialists, 
were unambiguous, measurable, observable, and feasible. (Some 
senior economists later became very proficient in- this.) in 
any case, from reviewing the work of the behavioral specialists, 
the economists reached the conclusion that behavioral objectives 
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prepared by non-economists could never be satisfactory. The 
primary reasons given by the economists were that the analysts 
often missed the point, often emphasized the wrong. things , 

I and continuously left important things out (i.e., there were 
j gaps in content coverage). 

i 

Approach #2 

As the number of economists on the project team grew, 
it became possible to initiate a new approach which, it was 
hoped, would produce behavioral objectives which the senior 
economists — particularly the faculty that taught the course — 
would find acceptable. Approach #2 -tailed for creating work 
teams consisting of one junior economist and one behavioral 
analyst, who would work under the cognizance of a senior 
economist. It was reasoned that the junior economist could 
help insure acceptance from the senior economists while the 
analyst would make sure that the behavioral objectives were 
properly written. 

The same problems that were found in approach #1 
quickly emerged in approach #2. There seemed to be the same 
inability to prepare "acceptable" behavioral objectives. 

In desperation, senior economists were preparing larger and 
larger descriptions of topics for use of the two-man teams, 
but the senior economists still found themse Ives unable to 
provide the terminal behavioral objectives that met the 
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standards of the behavioral analysts. Meanwhile , the junior 
economists who were paired with analysts learned to write 
fairly good behavioral objectives, in the judgment of the 
I behavior specialists. 

This approach also did not work. Attempts to "orient” 
the senior economists to behavioral objectives continued 
almost constantly, but with no apparent success. The failure 
of this approach led to approach #3. 

Approach #3 

This approach called for a senior behavioral scientist 
to be paired with a senior economist. Together, they would 
write the behavioral objectives for a topic, and both partici- 
pants had to agree on the specific objectives before they 
could complete a topic. The behavioral scientist's role was 
to guide the economist into producing or accepting the 
behavioral objectives that defined the senior economist's 
interpretation of the topic's goals cind scope of content. 

While the output of approach #3 was judged by the economists 
to be the best thus far, it proved to be extremely taxing 
on the participants and therefore could not be continued. 

While this approach was being discontinued, approach #4 
became possible. 

-39- 



O 




EDUCATIONAL TECHNOLOGY CENTER 



Approach #4 



In this approach, narrative descriptions of topics 
were virtually discarded. Instead, reliance was placed on 

/preparing behavioral "hierarchy charts" that defined topics. 

i 

I , 

| The concept of a hierarchy chart as used in approach #4 
evolved out of the failure of narrative information, both 
written and oral, to communicate among team members. At 
first,' the hierarchy charts were strictly the province of 
the behavioral analysts who used them in at tempting to define 
the structure of a topic. More specifically, after an econo- 
mist described a topic, the behavioral analyst would then 
attempt to translate this information into behavioral objec- 
tives, and he would arrange these objectives in a hierarchical 
fashion to indicate presumed relationships of the indicated 
competencies, including the dependency relationships, pre- 
requisite competencies , etc . 

Senior economists, whether with ETC or the economics 
faculty, had declined hierarchy charts because they did not 
convey, in their view, the full range of content required. 
Gradually, however, the charts became the working document 
of communication between senior economists and the analysts 
and between junior and senior economists. Soon the charts 
began to include the more detailed descriptions of content 
required by the economists. At the same time, the behavior 
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or behaviors that would demonstrate if a person has that 

knowledge were, as before, stated in the charts. As a result, 

i 

economists and analysts were both satisfied, because the 

charts communicated equally well with both groups. Since 

each box on a hierarchy chart included (a) description of 

content and (b) one or more behavioral statements, the charts 

+ * 

could readily communicate with people of widely different 
backgrounds, including administrators, instructors, psycholo- 
gists, and students. 

Prior to the use of approach #4, if an economist did 
not like the behavioral objectives for a topic, the analyst 
generally was blamed. As a result, the economists found that 
they agreed in not liking behavioral objectives produced by 
non-economists. Significantly, it was on content grounds, not 
'behavioral" grounds, that economists rejected the objectives. 

The emerging form of the hierarchy charts allowed 
economists to state in creating a chart for a topic, all of 
the ’’content" he believed was critical. And there Was no 
requirement placed on the economist that he must state the 
content in a format resembling a behavioral objective. This 
would come later, with help from a behavioral specialist. 
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The economists were asked to generate charts 
that sought to relate the content items in a hierarchical 
fashion. Thus, an economist's first-cut, content-only 
version of a hierarchy chart for a topic would tyoically 

I 

I have a pyramidal appearance, as indicated in Figure 4. This 

I "topic structure " proved to be an important prerequisite for 
the next step , specifying the actual behaviors. 




Fig. 4 - — ^op three levels of hierarchy chart 
in early stage of development. 
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A content-only hierarchy chart produced by an econo- 
mist might include twenty, fifty, or 100 content boxes. With 

t 

this preliminary version done,' the economist would be joined 
by a behavioral specialist. The role of the behavioral 
specialist was to work with the economist to (a) create one 
or more "behaviors" for each content-box, that would demon- 
strate that the student "possessed” that content, (b) insure, 
to the extent possible, that the organization of the chart 
reflected competency hierarchies, and (c) produce learning 
time estimates for each box on the chart ’(since the students 
had to be able to acquire the behaviors in the learning time 
available to them). 

This working relationship proved to be very harmonious 
between analysts, and . economists. Each chart went through many 
iterations before a balance was found between content, behav- 
iors , and amount of student time allocatable to a given topic . 
The primary strength of these charts was that they graphically 
described the status of the development of a topic's speci- 
fications at a given time. There was no longer the question 
of content versus behavior. Both content and behavior were 
required for every box included in a chart, and while the 
content was the primary responsibility of the economists, the 
analyst was held responsible for the quality and integrity of 
the behavioral' components of the chart. A chart served as a 
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constant visual referrent which reflected all decisions 
to-date for a topic. . As such, the charts came to symbolise 

I 

cooperation and results, and quickly became the primary 
working document and means of communication between the 
analysts and economists. 

As stated earlier, the senior economists always 
believed that they were in basic agreement with each other 
about what the topics should contain — that the lack of 
understanding of economics on the part of the analysts was 
the cause of the "acceptance" problem. The hierarchy charts 
made it possible for a senior economist and a behavioral 
specialist to mutually approve a chart. When such approval 
was given, formal behavioral objectives were then written 
for each box in £ chart. Such objectives were' carefully 
reviewed by the cognizant senior economist and others for 
conformance to the chart and for other characteristics of 
quality. However, when the behavioral objectives produced 
in this fashion. were reviewed by other senior economists — 
whether they were on the Center’s staff, were outside econom- 
ics consultants, or were faculty members at the Naval Academy 
— the objectives were typically criticized on the grounds of 
content. 

It gradually became clear to the non-economist members 
of the project* that no senior economist could happily or 
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comfortably accept completely the behavioral objectives 
produced by another economist. The extent of the disagree- 
ment among senior economists became apparent only when some 
senior economists, working with analysts, began to "sign off" 
on specific behavioral objectives. Thus, the conflict 
shifted to senior economist versus senior economist. 

This conflict was never completely resolved. However, 
it was capable of being controlled, and the hierarchy charts 
proved to be the basic means of obtaining this control. Once 
a senior economist agreed on how much student time was to be 
allocated to a topic, he had accepted a limit as to what the 
student could be expected to learn during that time. It was 
recognized that a student could not be expected to learn 
everything about' a topic, all other considerations aside. , 

The charts same into use as instruments of negotia- 
tion between the senior economists. Each box in a chart 
recorded the learning time, for the content-behavior it speci- 
fied. Once the total learning time in a chart equalled the 
amount of student learning time allocated to the topic, it 
was not possible to add something new to a chart without 
taking something else away. As a result, when a senior 
economist criticized a chart for not containing something, he 
was asked to indicate what he would eliminate from the chart 
so that his desired content could be added. 
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It soon became apparent to the senior economists 
that each would have to compromise on what he personally 
wanted to see in the charts . Behavioral specialists worked 
closely with the senior economists in effecting tradeoffs 
that the economists agreed upon. The charts became the 
documentation of the agreement, not only between the analysts 
and the economists, but also among the economists. When 
hierarchy charts had been agreed to in this fashion, general 
approval of the behavioral objectives produced from such 
charts became almost automatic. Approval of the charts 
became almost synonymous with approval of the behavioral 
objectives , 

When it was clear early in approach £4 that senior 

economists would - not accept each other's work, products, 

certain significant changes ware made in the project team 

organisation and in the decision-making structure. Strong 

behavioral specialists were moved into key positions above or 

equal to senior economists and they were given expanded 

decision-making authority . This change did not permit the 

behavioral specialists to dictate content or behaviors. 

Rather, it gave the behavioral specialists direct control 

over certain proceedings, including the handling of disagree 

ments among the economists. For example , ■ economists were 

constantly tempted to go back at id revise work done months 
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earlier — including work that they themselves may have already 
revised two or three times. Management-level behavioral 
specialists established hard and fast dates after which a 
work product was "frozen" (l.e., could not be changed except 
by authority of the Project Director) and otherwise exerted 
direct control over decision-making . 

This organizational change, coupled with the use of 
hierarchy charts, proved to be very effective* The apparent 
competitiveness between economist and economist seemed to 
subside. Behavioral specialists gained increased respect 
in the eyes of the economists, after having been viewed for 
so long as the cause of the problem. 

Approach £ 4 was used to complete the task of speci- 
fying the requirements for the new multi-media economics 
course. The approach appeared to be more successful the 
longer it was used, as, team members became experienced in it 
and made refinements. 

Summary of Successful Approach 

In summary, a satisfactory-, workable method for 
establishing educational behavioral objectives and for gaining 
inter-personal and inter-group agreement on them evolved in 
the project. The elements of this method were as follows: 
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1. General goals for the course were specified. 

2. The instructional topics to make up the course 
were listed. 

3. For each topic, a statement of key points of 
emphasis and desired areas of content and 
behavior was prepared as a guide for direction 
and scope, 

4. Maximum student time was determined and allocated 
to specific topics. 

■5. A "hierarchy chart" was prepared for each topic, 
and each box in such a chart included (a) a 
statement of course content and (b ) a statement 
of one or more behaviors that would demonstrate 
an understanding of that content. 

6. The hierarchy chart for a topic was used as the 
medium for negotiating differences among 
decision-makers and for documenting the final 
forms of the " cipproved" learning requirements 
for the topic. 

7 . Formal behavioral objectives were prepared in 
conformance to the hierarchy charts. 
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CHAPTER V 



PREPARE TEST ITEMS 

There were no major problems encountered in the 
preparation of test items. The task was performed very 
similarly to the way in which it was planned. There was, 
however, a significant rationale employed in task performance 
that , it was concluded, played an important part in the suc- 
cess of the task . This rationale deserves particular re f srence . 

Rationale for Test Item Design 

It was established early in the Project that a criter- 
ion test item was to be essentially a "mirror image" of the 
behavioral objective for which it was prepared and it had to 
be an objective measure. In other words, the exact criterion 
behavior specified in the behavioral objective was to actually 
be a part of the test item. This had the effect of forcing 

i 

specificity in the behavioral objectives. Every criterion 
behavior stated in the behavioral objective had to be explicit, 
measurable, feasible, unambiguous , etc. , because that behavior, 
perhaps exactly as stated, was the "answer" to the associated 
test item. All behavioral objectives, as a result, provided 
"answers" (i, e .f very explicit descriptions or examples 6f 
criterion behavior) which were to be used. in the associated 
test items or used as models for similar responses. 
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The precision required in no way meant that test items 
could not sample a behavior that applied to a class of condu- 
ctions. For example, a behavioral objective might state that 

the student will be able 11 to solve equations of the following 
type," under specified conditions. Examples of the- type ef 
equation would have to be provided (with a description of the 
"correct response" to each such example), along with an indi- 
cation of the minimum acceptable level of performance and a 
statement of the conditions under which the student would have 
to perform. In such a case, test items would be modeled after 
the sample equations and other ' information in the statement of 
the behavioral objective. 

Another advantage of the "mirror image" approach 
concerned the content validity of the test items — in other 
words, with the question of whether or not the test items 
were measuring the same behaviors as called for by the behav- 
ioral objectives. This approach enabled "face validity" 
judgments to be unequivocally made by subject matter experts, 

*1 

■‘■Teaching to behavioral objectives should not be 
confused with "teaching to the test." It is" simple to 
illustrate the difference. A behavioral objective may 
specify that the student will be able to add twe or more 
single-digit numbers. A student who achieved this objective 
will be able to add 1 and 9, 2 and 4, 6 and 3, or any 
other combination of single digit numbers. In "teaching 
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Procedural Considerations 



i 

There were, of course, minor procedural changes which 
took place. The preparation of the first test item at the 

time the behavioral objective was being prepared was. considered 
to be a very important procedural modification. This seemed to 
make the process of establishing behavioral objectives a little 
easier , by helping to establish the nature of the criterion 
behavior. (in other words, some people can write "good" test 
questions easier than they can write "good" behavioral objec- 
tives. They may prepare test items as a way of "exploring" 
the kind of behavior that the objective should call for. When 
they "recognise" what is required, they may then write a 
behavioral objective that represents a class of feasible test 
items . ) 



Three or more criterion test items were prepared for 
each performance objective. (Figure 5 illustrates a performance 
objective while Figure 6 gives a criterion test item for the 
objective.) A criterion test item was intended to answer 



to the test," the student would learn to add only the single 
digit numbers that appeared on the test. The student might 
be able to add 2 and 4 but unable to add 2 and 3. 

A pool of criterion-referenced test items was pre- 
pared for each behavioral objective included in the Multi- 
Media Economics Analysis Course. Each such test item 
represented a "sample" of the desired, criterion behavior. 
From the pool, test items were drawn to construct pre-tests, 
post tests, and imbedded tests. These tests were used to 
determine whether or not the student possessed a specified 
behavior . Instruction is aimed at behavioral obj ectives . 
Tests merely measure whether or not objectives are met. 

-51- 



13 DUC ATIONA Ij TEC □ "NGIjOGY Cl-lNTBIt 



the question: Has the -student achieved the related learning 
objective? The items yielded a yes or no ansv7er for each 
objective. Test items were subjected to essentially the same 
type of review process as performance objectives. The test 
items had to be reviewed and approved by several subject 
matter experts and a tests and measurement specialist before 
they were permitted to go into the test item pool. They were 
also subjected to revision based on student tryout results. 
Test items were used to construct pre-tests , post- tests, 
practice tests , and learning-imbedded tests , all of which 
are elements of the multi-media course. 



- 52 - 



O 

ERIC 



I3DUOAT IONA f j T ncliNOTJOHY CENTKl* 



2753 



Conditions : 



Given alternative graphs concerning 
total demand for money and asked 
which graph shows the total demand 
for money as a function of interest 
rate for a given level of* income/ 

Performance : TJie student will select a graph 

equivalent to the following: 




Criterion: 'Implicit 



Fig. 5. — Example of a Performance Objective 
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2753 A Which graph shov:s the correct configuration for 
the total demand for money as a function of 
interest rate for a given level of income? 






ANSWER: b 



Fig. 6. --Example of a Criterion Test Item 
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CHAPTER VI 



SEQUENCE INSTRUCTIONAL ACTIVITIES 



The "sequence of instructional activities" refers to 
the order in which students interact with units of content. 

The literature regarding instructional sequencing demonstrates 
that there are no conclusive data on how it should be done.'*' 
Heimer, after reviewing sequence theory development, concluded: 

Adequate teaching alqori thras which specify the steps 
to be taken in order to construct an instructional 
sequence in the presence of a given set of educational 
ends and a given set of circumstances and with some 
assurance of efficiency, do not exist. ^ 



Despite the lack of agreement in the literature on 
how sequencing should be approached or performed, little 
difficulty was encountered conceptually or operationally in 
the task performance . 



1-See, for example, David J. Klaus, "An Analysis of 
Programming Techniques , " Teaching Machines arid Programmed 
Lear n.i ng , lit Data and Dir act io r. s , ed. by Robert Glaser 
('Washington, D.C.: Department of Audiovisual Instruction, 

1965), pp. 141-143. 

Ralph T. Heimer, "Conditions of Learning in 
Mathematics: Sequence Theory Development, " Re view of Edu - 

cational Research , XXXIX, No. 1, 1569, 506. 
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The Macro-Planning Chart (described in Part III) , 
prepared in connection with allocating student learning time 
to instructional topics, proved to be an effective first step 
in sequencing the instruction. The MPC provided the initial, 
"general" sequence of the topics expected to be included in 
the course. 

An effective second step in sequencing was the pre- 
paration of the hierarchy charts, with their great emphasis on 

entry-level behaviors , intermediate behavioral objectives , and 
terminal behavioral objectives. At the point where all hierarchy 
charts were "approved,” the entire course could be viewed 
as sequenced in a hierarchy of competencies, from the entry- 
level competencies to the final competencies to be taught in 
the course. 

Sequencing Concept Areas 

As would be expected, the relationships among com- 
petencies to be learned (behavioral objectives) were not 
linear. Rather, at the concept area level as well as the 
topic level , the relationships resembled a series of pyramids . 

The four major "pyramids" of objectives represented the four 
major concept areas and were related in the fashion illustrated 
in Figure 7. Note how the pyramids , or major concept areas, 
thems elves are hierarchically related. It was necessary, it 
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was judged, for a student to achieve the objectives in Concept 
Area 1 (pyramid 1, Figure 7) before he could begin (possess 
the entry behavior for) either Concept Area 2 or Concept Area 3 

(pyramids 2 or 3). Similarly, it was thought that the student 
had to have achieved the objectives in both Concept Area 2 and 
Concept Area 3 before he was qualified to begin Concept Area 4. 




Fig. 7. — Major Hierarchical Groupings of Objectives 
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In terras of the 
could study concept areas 
Figure 8. 
as appropriate 
ed for use 

Concept Area 
Sequence "A" 
(Ascending order) 

4 

3 

2 

1 

Fig. 8 - — Alternative 



presentation, then/ a student 
either order shown below in 

sequence was' theoretically 
"A" (Figure 8) was select 
the period of the 

Concept Area 
Sequence "B" 

(Ascending Order) 

4 

2 

3 

1 

Sequencing of Concept Areas 



course 
in 

While either concept area 
as the other. Sequence 
in presenting the course during 



project. The decision not to allow both sequences (i.e./ so 
the student could elect his preference) was made to avoid 
possible problems during the preliminary presentations of the 
course. In particular/ there was the concern that no avoidable 
risks be taken in administration while materials were under- 
going preliminary testing and validation. The intent was to 
prevent contaminating the mate rials “tryout results with 
negative effects of operational problems. 



O 
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Sequencing Topics 



As suggested by tile pyramids in Figure 7, each concept 
area consisted of hierarchically-related topics. Many possible 
| sequences of topics existed for each concept area. Again, for 

r 

| very practical considerations, decisions were made to select 
only certain sequencing alternatives for use in the course. 

It was concluded that it would not be feasible to provide all 
possible topic-sequencing options, considering media-materials 
logistics, student management, and other factors. The same 
concern with avoiding risks of administrative problems during 
materials testing also prevailed. 

Sequencing remediation 

The decision was mads during pre-contract planning that 
the function of remedial learning was to support core learning. 
Two types of remediation V7ere considered possible, "prerequi- 
sites" remediation and "regular" remediation. 

Prerequisites remediation had to do with assumptions 
about entry behavior . Let us say that we assume in the cours e 
that all students can use graphs in some specific way. Through 
a pretest, "we may find that some few students cannot use graphs 
as assumed. A "prerequisites" remediation topic (e«g«r "Using 
Graphs") might be necessary fer this situation. L 
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Regular remediation had to do with learning problems 
encountered by students who apparently had the prerequisite 
behaviors. An example might bo the case where a student had 

jdifficulty in solving a problem of a given type. Me may be 

I 

| referred to a "regular" remediation package that teaches the 
desired problem solving behaviors in a different style or at 
a lower level of difficulty than in the original learning 
activity. 

The Macro-Planning Chart had set aside an average of 
about three hours per week for prerequisite or regular reme- 
diation. However, no remediation "topics" of any kind were 
assigned to the segment buckets in the Macro -Planning Chart. 

It was believed to be impossible to judge in advance, v/ith 
any acceptable degree of accuracy, either the prerequisite or 
the regular remediation requirements. It was decided that the 
more cost-effective approach would be to empirically determine 
the remedial learning requirements. 

* 

While no remediation topics were specified, a remedia- 
tion option was left open following each pretest (prerequisite 
remediation) and each post test (regular remediation) . 

Sequencing Enrichment 

Enrichment learning objectives were not, in the main, 
defined in a separate task from determining the core learning 
objectives. As described earlier, it proved to be very diffi- 
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cult to keep the size of the course r as implied by the behav- 
ioral objectives specified, within bounds. It was estimated 
that sufficient behavioral objectives for a five-semester 
economics course were generated. About two semester's worth 
jwere selected to be the "core" objectives of the Multi -Media 
'Economic Analysis Course. * Many enrichment objectives were 
selected from the remainder. Other enrichment objectives were 
developed specifically as "enrichment." 

In the MMCD project, the behavioral objectives were 
stated at a very specific level, requiring perhaps ten minutes, 
in most cases, for a student to achieve the objective, assuming 

he had the prerequisite competencies. It was not possible to 
pick this isolated objective and that isolated objective, etc., 
and call them "enrichment." There had to be a valid "behav- 
ioral path" to every objective in the course, which meant that 
all prerequisite competencies to achieving an objective, whether 

it was core or enrichment, had to be provided. Thus, enrich- 

* < 

ment objectives also had to form "instructional topics," and 
either a core segment or another enrichment segment provided 
the entry behavior for a given enrichment topic. As With 
core topics, an enrichment topic could require one or more, 
segments. The sequencing of the enrichment was based on such 
dependency relationships. 
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Sequencing Other Activities 

The sequencing activity also included determining when 
pretests and post tests should be given, when counseling should 
be available, and what options existed for the student at 
specific points in the course. Sequencing decisions were 
recorded in the form of a "Micro-Sequencing Chart" for each 
concept area. Figure 9 shows such a chart. 
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In the Micro-Sequencing Chart shown in Figure 9, the 

first segment (Cl) was a group orientation session. The 

second segment (C2) Weis a pretest, the third segment (C3) was 

a counseling session, when pretest scores and learning con- 
i' 

I tracts were discussed, the fourth (C5) was a self-instructional 
! segment, and so on. If the student progressed at an acceptable 
rate , when he passed the post test of segment C3.1 , he had an 
option of taking certain enrichment segments or continuing on 
to more core segments. 

No matter how far a student progressed in the course, 
as long as his rate of progress was acceptable , he could 
exercise his option to take any enrichment material he had 
earlier decided not to take. This "continuing option" is 
not shown in Figure 9. Also, neither the "student-generated" 
enrichment learning activities nor the "instructor-generated" 
enrichment learning activities are represented there. Such 
enrichment was to be generated during the course, to be respon- 
sive to the special needs or desires of groups of students or 
individual' students . This type of enrichment tends to revolve 
around issues that are currently making headlines, which the 
students perceive as "relevant." 
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CHAPTER VII 



SELECT THE MEDIA 

Very little lias been produced in the past fifty years 
concerning when to use what instructional media. In his review 
of instructional design and media selection factors Saettler 
points out that the predominant type of media study conducted 
lias been the media comparison study, usually yielding a finding 
of no significant difference.^- He concurs with the observations 
by Knowlton that these studies were not actually research on 
media because their experimental designs did not provide for 
separating the physical characteristics of the media from the 
sign vehicles of the message they carried. 2 Lumsdaine and May 
in their re vie w o,f media research have stressed the inherent 
limitations of such media compar ision studies. ^ 

ipaul Saettler, "Design and Selection Factors" Review 
of Educational Research 38: 115-128; April 1968. 

^ James Q. Knowlton, "A Socio- and Psycho-Linguistic 
Theory of Pictorial Communication" (Bloomington: Indiana 

University, Division of Educational Media and Audio Visual 
Center, 1964) . 

3a. A. Lumsdaine and M. A. May, "Mass Communication 
and Educational Media," Annual Review of Psychology , XVI, 

(1965), 475-534 , 
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A monograph by. Briggs and others reviewed media 
research and commented in some detail on its shortcomings.^ 

t 

Studies were classified into (1) comparative effectiveness 
studies, (2) utilization studies, and (3) basic studies, for 
each of seven classes of media. As to the value of the 
research findings in making specific media choices, it was 
concluded that "neither laboratory nor media studies has 
provided complete enough information to eliminate the uncer- 
tainties involved in making such choices ... neither the learning 
psychologist nor the classroom teacher can justify such 
decisions entirely on the basis of present research evidence. 

A significant step in the direction of developing a 
comp r eh ensive and systematic media selection procedure is seen 
in the work of Gagn/ and Briggs. Gagne developed a hierarchical 
model of eight types of learning, and defined the particular 



^Leslie J. Briggs. , et al . Instructional Media: A 

Procedure for the Design of Multi-Media I n struction, A Critical 
P.eview of Research> and Suggestions for future Research, 
Monogram No. 2 (Pittsburgh: ~ American Institutes for Research, 
1967) . 



Plbid. , pp. 137-138. 
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conditions (of learning) found necessary to facilitate each 
type.® Media are to be selected which can best produce the 
conditions of learning required. It is assumed that different 
media would be required to produce different conditions of 
learning. 

Briggs developed a procedure, based on Gagne's model, 
which called for analyzing behavioral objectives to identify 
the types and conditions of learning involved, determining the 
instructional events that would provide the desired conditions 
of learning, and then matching the characteristics of the 
instructional events with the characteristics of media. The 
procedure described by Briggs was attempted in the Multi- 
Media Curriculum. Development Project. 



6 Robert M. Gagne, The Conditions of Learning (Mew York 
Holt, Rinehart and Winston, Inc., 1965"). 

"^Briggs , et al. , Instructional Media . 

-67- 



educational tecijnologv cisntkh 



Problems Encountered 

The original media -selection plan was found to have 
deficiencies as applied in the project. It may be- of interest 
to summarize the problems encountered before describing the 
new plan that emerged. 

The effect of using the media-selection plan was to 
cause media that were "not preferred" to be eliminated from 
consideration, while leaving a group of acceptable media or 
media-mixes from which the final selections had to be made. 

In the media-selection analysis, however, an unacceptable 
medium or media-mix could become acceptable simply by adding 
a supplementary medium or otherwise changing the media-mix. 

Many permutations were possible. 

There was always the question of where to draw the 
line in terms of different mixes that might be considered. 

For example, assume a CAI carrel included a cathode ray tube, 
a slide projector, an audio output device, a hardcopy output 
device, a keyboard input, a light pen input, and some 
pre-printcd material for the student. Such a carrel alone 
could result in over one hundred different possible media-mixes 
Thus,, there was always an ample supply of media combinations 
that could provide satisfactorily for particular conditions of 
learning. 
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Another factor in the media-selection analysis was 
the homogeneity of the learning conditions that seemed called 
for throughout the various parts of the course. This seemed 
to be the logical result of the great similarity in the 
characteristics of the content and what students were to do 
with it (i.e., the behaviors) from segment to segment of the 
course . Accordingly , .it was possible , if one chose to do so , 
to select a single media-mix, from among the alternatives 
available, for use in rather lengthy sequences of instruction. 

It also became clear that there had to be "trade- 
offs" made throughout the course, since resources are not 
unlimited. For example, media "A" may cost twice as much as 
media "B" (actually , the cost ratio can be fifteen or more to 
one), and .it will- be "better," we judge, than media "B. " 

Questions arose, such as: (1) regardless of value, is there 

enough money available to even consider media "A"?, (2) how 

much "better" does a medium have to be to justify paying 
twice the cost (and what criteria, with what reliability, 
should be used in making the decisions), and (3) should resources 
be evenly devoted to mediating the different parts of the course 
or is it better to concentrate resources on mediating a portion 
of the course, to produce the greatest average learning. 
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Still another factor was that the plan seemingly did 
not give sufficient emphasis to the administrative-operational 
setting in which the course would be presented. All told, it 
appeared that the original media selection plan focused largely 
on the theoretical aspects of learn ing, somewhat at the 
expense of environmental factors that can have a major effect 
on the learning outcomes. 

The New Scheme 

It was concluded that the media selection plan orig- 
inally proposed did not go far enough. In general, it seemed 
that tiie scope of variables implied by the original plan was 
too narrow, and there was no mechanism to weight the vari- 
ables in terms of- their relative importance. As a result, . 
the media selection approach underwent a revision. A two- 
step media selection plan emerged. 

Media selection criteria ware derived from the variables 
which would determine media "effectiveness." One class of 
criteria was derived from the requirements of specific 
learning situations, and these may be called the "micro" 
criteria. A second group of criteria, often overlooked, was 
derived from the general environment in which the learning 
situations were to be provided. These may be termed the 
"macro" criteria. 
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Step 1. Micro-Criteria Analysis 
There were several steps in this analysis, which was 
directed toward determining which classes of media could be 
used in particular instructional sequences. This analysis 
assumed that performance objectives had been explicitly 
defined and sequenced and ; that the sequencing (or sequences) 
reflected hierarchies of competencies called for by groups of 
objectives. The analysis began by examining each individual 

objective in a sequence to determine the basic requirements 

* 

of tlie learning called for. This analysis sought to determine 
the specific stimulus requirements, response requirements, and 
other requirements for producing the desired learning. 

The ore t ically , these requirements for learning , as 
determined by this analysis, had to be provided for through, 
the instructional strategy, if the learning was to be efficient 
or to take place at all. The major parts of an instructional 
strategy included (a) the characteristics of the message to be 
presented; (b) the characteristics of the possible responses 
from the student, including the desired response; (c) the 
requirements for receiving and analyzing the adequacy of the 
response; (d) the characteristics of the feedback to the student; 
and, finally, (f) the technique of programming all of these. 
Categories (a) through (d) above were the -"requirements for 
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learning" that the micro-analysis sought to determine, since 
they established what capabilities the selected media must have. 
In Mother words, they produced the micro-criteria for use in 
the media selection process. Figure 10 gives examples of 
the kinds of micro-criteria that may be produced. 

The result of the micro-criteria analysis was a 
statement of the media capability requirements. (A distinc- 
tion was made between the absolute requirements and the "nice- 
to-have" capabilities — an attempt was made to never confuse 
the two . } 

A list of types of media feasible for consideration 
in the learning program was compiled. The major characteris- 
tics of classes of such media were then delineated. At this 
point, it was possible to compare the media capability require- 
ments with the characteristics of media feasible for consider- 
ation. This produced two groups of media, one that was 
"acceptable" and one that was "unacceptable," based on 
attempts to match requirements with capabilities. This 
concluded the micro-criteria analysis. 

Before describing the next step, a comment is in order. 
The more logically appealing media selection approaches call 
for an analysis that is conceptually similar to that described 
above. Some specify that an analysis of great depth and 
detail be conducted, while others are much less demanding. 
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A. Stimulus Characteristics 



D. Media Function 



1. Visual 



1. Present stimulus 



a) Black and White 

b) Color 

c) Still 

d) Motion 

2. Aural 

a) Voice grade 

b) Hi-fi 

3. Referability 

4. Three-dimensional 

5. Coding complexity 



2 . Direct attention 

3. Provide model of 
■ expected perfcr-r 

mance 

4. Furnish external 
prompts 

5. Guiding thinking 

6. Inducing transfer 

7. Assessing attain- 
ment 



B. Response Characteristics 

1 . Spoken 

2 . Written 

3. Covert 

4. Practice 

5. Creative 

6. Time factors 

C. Level of Abstraction 

1. Real 

2 . Synthetic 

3. Symbolic 



8. Providing feedback 

E. Type of Objective 

1 . Core 

2 . Remedi al 

3. Enrichment 

F . Learning Situation 

1 . Group 

2 . Individual 



Fig. 10. — Micro-Criteria For Use in Media Selection 
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All of these approaches, though, are trying t.o make some sort 

of effective match between the learning conditions and the 

/ 

media characteristics. But it is at this stage in. the media 
selection approach that the process seems to break down. 

Having isolated a group of media which at least theoretically 
could meet the requirements for a given instructional sequence, 
it would appear that selections could now be made, taking into 
account basic considerations of price and durability, and that 
one could be confident of the result. The result, in fact, 
could be failure of the media strategy and the learning program 
that used it. 

Approaches which go little or no further than what has 
been termed the micro analysis are essentially seeking to 
optimize at the subsystem level. (In other words, much like 
strengthening one link in a chain without knowing how strong 
the other links are.) Such approaches court failure by ignor- 
ing or only making passing mention of the general environment 
in which the learning is to be provided. To ignore the broad 
context or general environment, from which important media 
selection criteria may be derived, is to seriously endanger 
the success of any media selection scheme. The next step 
dealt with such environmental considerations. 
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Step 2: Macro-Criteria Analysis 

Macro-criteria were derived from elements of the 

f 

j 

general environment in which the learning was to be provided. 

The relationship between the micro-and macro-criteria and how 
they were derived is illustrated in Figure 11. The figure also 
shows that micro-criteria produced candidate media, but the 
final selection from among the candidates was done using the 
macro-criteria. The first major consideration in deriving 
macro-criteria was the "level of concern f " since it was a 
basic requirement in identifying the goals, resources, and 
constraints that could affect media selection and utilization. 

Level of Concern 

The level of concern for media decision-making may be 
a single concept, a single lesson , a complete instructional 
unit, a complete course, or for that matter , a department or 
an entire school. The higher the level, the broader the goals 
are, the more resources that can be applied, and the fewer 
constraints that cannot be relaxed. Goals of the school have 
an impact on the goals of the division, and the division’s 
goals give general direction to the department's goals, and so 
on. Goals determine how resources will be allocated and, there- 
fore, they influence the . constraints that will pertain to the 
media program. .Formal and Informal’ administrative policies and 
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Feasible Media 



i 



Behavioral' Objectives 

i 

Tvpe of Learning 

1 

Learning Objectives 

1 

Stimulus-Response 
Performance Analysis 

I 

Micro-Selection <* 
Criteria 



s- Media 

\ Characteristics 



Goals, Resources, 
General Strategies , 
and Constraints 



Candidate 

Media 

i 

'i 

Candidate Media 
Characteristics 



-s* Macro-Selection 
Criteria 



V 



Selected Media 



Fig* 11. — Media Analysis and Selection Process 
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practices reflect goals , and the success of the media program 

is going to depend in large part on how it supports the goals 

| 

of 'the next higher level. Obviously, if a planned media 
program for a department is at odds with explicit or implicit 
division .policy , it is highly probable that the program will 
encounter some significant obstacles. 

There is little value, for example, in someone specify- 
ing media for a self-instructional program if this type of 
program is unacceptable at the school in the first place. 
Conversely , one may find that he is selecting media for a 
demonstration project which could lead to an entire school 
system adopting a self-instructional course in a given subject. 
The policy on expansion and dissemination of the results of a 
media selection-utilization project should be clearly under-r 
stood.. 

Being attentive to the level of concern in the MMCD 
Project helped to determine . the written and unwritten policies 
and goals within which the media program would operate. To 
the extent possible, all pertinent policies and goals were 
made explicit for purposes of media planning. In some areas, 
there was no identifiable policy, and the media planner had 
an opportunity to influence its formulation at the appropriate 
time . 
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Clearly identifying the resources available for. media 
utilization was equally impox'tant. The resources available 
were often not immediately apparent. It was important to 
idif ferentiate between resources required to initiate. a media 
! strategy and the resources required to operate it. (If 

i 

materials are going to be’ produced for sophisticated equipment, 
for example, then a great deal of funds may be required to 
provide the program.) 

Some media programs can be very costly to maintain 

* 

(e.g., those using computer-assisted instruction) . Types of 

resources that were considered include the following: 

. Funds available~“v.’hat sources, how probable that they 
will actually be provided, what dates will they be 
available , etc . 



. Personnel avail able --who a ire they, can the release 
date be guaranteed in writing, what are their 
interests and capabilities, part-time or full- 
time, etc. 

. Time available--when is the media program required, 
what target dates must be met, etc. 

. Hardware available- -what media are now available here 
or from elsewhere, what is lease price versus 
purchase price, what information services and other 
services are available for the media, are special 
skills required to operate the media, what are 
operational capabilities, procedures, and costs, etc. 

. Software? available — what materials are available 
here or elsewhere for the different media, at what 
cost and with what quality, what materials would be 
useful if modified, etc. 
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Constraints are more or less the opposite side of the 
resources coin. While the lack of a needed resource in 
sufficient quantities is a constraint, it was also important 
not to take certain things for granted and, as a result, fail 
| to uncover significant obstacles. One may be interested in 
i innovative programs, for example, but the rest of the faculty 
might like the current approach . 



Programming Strategy 

Another key source of macro-criteria was the instruc- 

* 

tional programming strategy. It was almost impossible to 
consider programming strategy without classes of media coming 
to mind. Once a decision was made on programming strategy, 
though, some media were automatically ruled out. Similarly, 
once the media were selected, certain programming strategies 
could no longer be considered. For example, computer-based 
simulations were selected to demonstrate to students how 
various concepts and principles relate to each other in an 
econometric model under conditions which the student himself 
may vary. Many variables may be handled and the relationships 
among them may be extremely complex, yet. the model can be 
easily and quickly manipulated by the student at a computer 
terminal. This programming technique might never have been 
considered by someone completely ignorant of computer capabili- 
ties. Thinking .imaginatively about how specific media could 
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be used helped to develop new programming approaches, and as 
many alternative approaches as pos sib le were generated and 
considered before the final selection was made. 

An important point is that all programming approaches 
considered had to be compatible with the stimulus-response- 
feedbadt requirements that grew’ out of the analysis of the 
performance objectives. And of equal importance was the 
requirement that they reflect the characteristics of the target 
population of students. Alternative programming approaches 
had to be analyzed in relation to these and other factors 
const! tuting the overall instructional strategy . The various 
factors, of course, were interactive, so the selection of the 
final "configuration" of factors was done through an iterative 
process. This process was directed toward meeting the perfor- 
nance .objectives in the most cost-effective manner and within 
the framework of the policies and goals that prevailed. 

Ins tructor-Student-Media Interactions 

The importance of defining selection-utilization 
criteria that will generate acceptance of the new media 
strategy has been demonstrated by the number of media programs 
which have failed in their initial stages because the faculty 
or students involved did not like them. The matter of 
instructor-student-media interaction, which is the primary 
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basis for acceptability or noil-acceptability, was dealt with in 
the final step of the macro-analysis. It was a critical part 
of the overall instructional strategy. 

l Acceptance Interrelationships . --It was very important 

jnot to ignore that students and teachers have opinions and 
attitudes — sometimes very strong ones — regarding the appropri- 
ateness of various instructional media. The grades that a 
student has been getting will influence his media likes and 
dislikes. (The ghetto child, for example,- may have entirely 
different perceptions of specific media and the role they 
should play than the suburban child.) Also, the teacher will 
have his own criteria as to what constitutes "acceptable" 
media. He may believe that the purpose of media is to help 
him to teach better and be concerned about attempts to "replace 
the teacher" in some media applications. Or he may be eager 
to experiment with new approaches and new roles. Teachers 
will differ somewhat as to viewpoint, but as a group their 
attitudes toward a new media program can grow more positive 
or more negative, depending on how they are personally affected 
A new media program involves the students and the 
teachers. If the nature of this involvement is disliked by 
either party, the success of the program is jeopardized. If 
the students do- not like the program, negative comments will 
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soon be heard from the teachers. Tind, certainly, if the 
teachers are unhappy in their roles, this attitude is highly 
likely to pervade the student group. It is important , then, 
to take account of student variables which can promote accep- 
tance on the part of the students and define a role that is 
acceptable to the teacher . 

S tuden t V a r 1 ah ,1 c s . — Many student variables entered into 
the planning and decision-making for a media strategy. Many 
of the same variables important to the design of instructional 
messages were considered because it was necessary to avoid 
media that could constrain or otherwise affect the message in 
undesired ways. Media which did not distort the message but, 
rather, facilitated its delivery under the required conditions 
were preferred. 
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Instructor Variables .—The instructor variables were 
of a somewhat different kind but were considered equally 
important for a new program to be successful* The primary 
question was what role the instructor would play in a new media 
strategy, and this, it was thought, would depend on many 
factors. The following were considered: 

* Iiow much total time would the instructor spend with 
(a) students individually and (b) groups of students? 

. What activities would he be engaged in, as a percent" 
age of the total time? 

* 

. Would his role have more prestige and/or more pay 
than the conventional role? 

. Are the instructors qualified to fulfill the 
requirements of the role? 

. Can the instructors learn any necessary new skills 

under the conditions {time, money, etc.)? 

* 

. Will there be an increase or decrease in the overall 
workload of the instructors? 

. How much freedom and decision-making authority will 
the instructors have in the new program? 

. How much confidence clo the instructors now have that 
a proposed student-media-materials logistics plan 
will be adequately supported by the school adminis- 
trators? 

. What is the media history of the school (©• g* , have 
media been available and have they been used)? 

' This list could be made quite lengthy, but it should be 

sufficient to illustrate - some factors considered to be impor- 
tant in gaining, teacher acceptance. 
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Any proposed media program can cause anxiety among 
teachers who are uncertain as to how they might be affected, 

l 

particularly if self -instructional media are involved. It is 
understandable that a nev; media program may be seen as a major 
threat by the teachers and be in various ways resisted until 
an acceptable role for teachers has been defined and communi- 
cated to the teaching staff. 

Utilization of Macro-Criteria 

The macro-criteria that were derived in the MMCD 
Project are summarized below. (In another situation, other 
macro-criteria may be more important.) In reviewing these 
criteria, it is easy to recall media that might be acceptable 
or unacceptable in terms of a single criterion.' Certain media, 
however, will meet a given criterion better than some others, 
or for less implied costs than others. 

. Self-pacing — high priority must be placed on_students 
proceeding at their own pace. 

. Self-management — within broad limits, the student 
should be able to manage his own time and exert 
control over his own learning situation. 

. Cognitive structure — it is desirable to be able to 
demonstrate patterns and inter-relationships in 
problem-solving, real-world simulation circumstances. 

. Motivation-- the management of motivation should be 
enhanced by media (i.e., the media must reflect 
student characteristics). 
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. Administration -operation — the media strategy must 
fit within the administrative and operational 
constraints imposed by the environment. 

. Acceptability-the media strategy must be acceptable 
to the faculty who will be affected and to the 
students for whom it is intended. 

. Implied costs — the implied costs of the media 

strategy, for both development and utilization, must 
be within the reach of available resources. 

. Novelty --the novelty effect of certain media or 

medi a sequences should be exploited (a simple change 
of the media is often ref resiling and motivating to 
the student) . 

. Sequencing — media usage by the student and transitions 
from one medium to another must appear unhurried, 
smooth , and rational to the student . 

The macro-criteria that should be employed will vary 
from situation to situation, simply because the variables will 
differ between situations. This is important, since what is 
desired is a media program that will meet the requirements of 
a given s i tua tion . The list of speci f ic variables that coul d 
be important at the macro level could be quite lengthy... 

In the MiMCD Project, care was taken to properly weight 
the variables in accordance with their probable contributions to 
media effectiveness. For example ,' "acceptability" of the media 
strategy to faculty and students was considered itical in 
the MMCD Project, and selection criteria required to insure 
such acceptability was heavily -weighted, 
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CHAPTER VIII 



PREPARE MATERIALS 



in a review of the research on curriculum materials, 

i 

Popham distinguished between projects that produced admonitions 

&nd projects that produced materials, stating: 

To state it bluntly, it takes more than admonitions 
from curriculum seers, even if they are accompanied by 
polished curriculum guides, to alter the procedures of 
busy educational practitioners. The educational reformer 
who eloquently urges classroom teachers to change their 
practices may receive the accolades of the educational 
community, but the educational reformer who provides a 
set of usable curriculum materials for the teacher is 
more likely to modify what goes on in the classroom. 1 

The preparation of materials concerned four 
critical activities: 

1. Developing the presentation-feedback strategv 
for each media-mix selected 

2. Generating materials specifications 

3. Producing the materials 

4. Trying out the material with students and revising 
it as indicated by an analysis of the empirical 
results . 

These activities had to be effectively performed within 
hard and fast time limitations. On a specific day, the semester 
would begin, the students would arrive, and the Multi-Media 



iw. 

Educational 



James Popham, "Curriculum Materials," 
Research , XXXIX, No. 3 (1969), 319. 
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Economic Analysis Course had to bo "ready" for them. "Ready" 
meant that confidence in its instructional benefits had 
already been obtained through preliminary testing.* This, in 
turn, meant that one to two hundred student-hours of .material 
had to be generated and reach a "pre-validatcd" form during 
about a three-month period. 
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Presentation Strategies 



/ It was decided that Gagne^' s theoretical descriptions 

of conditions of learning for types of learning would serve as 
general models for the presentation strategies of the media. 
This meant that, as a first step, the type of learning called 
for by each behavioral objective in an instructional segment 
had to be determined. Each "segment of objectives" was care- 
fully reviewed and each objective labeled with an identifica- 
tion of one of Gagne's eight types of learning. 

The great preponderance of objectives was judged to 
be Type 8 (Problem Solving) and Type 7 (Principle Learning) , 
the two highest types of learning according to Gagne's hier- 
archy. Type 6 (Concept Learning) was third in frequency of 
usage and Type 5 (Multiple-Discrimination Learning) was fourth 
None of the objectives .was judged to be Type 1 through Type 4. 

For purposes of materials development specifications, 
it proved to be difficult and impractical to differentiate 
between Principle Learning and Problem Solving. In other 
words, it was difficult to find general criteria for the 
materials developers to follow which would adequately differ- 
entiate between how materials should be developed where 



^ Robert M. Gagne, The Conditions of Learning (New York 
Holt, Rinehart and Winston, Inc., 1965). 
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Principle Learning is called for versus where Problem Solving 
is called for. • There is as much variability in learning within 
each of these two types of learning as there is between the two 



types . 

There are "simple" principles and there are "difficult" 
principles. Similarly, there are "simple" problems to solve 
and there are "difficult” problems to solve. The amount of 
guidance required can vary within either type of learning. 

Gagne" anticipated some difficulty of this type when he stated: 

* 

In a fundamental sense, the capability learned by 
problem solving (Type 8) is no different from that which 
may be learned bv 'combining subordinate principles in 
the manner described as Type 7 learning. Both hinds of 
learning result in the establishment of higher “order 
principles. The two hinds of: learning appear to differ 
only in the nature and amount of guidance provided by 
verbal instructions . 3 

As a result, the . materials specifications combined 
Types 7 and 8 learning . into one explanation of conditions of 
learning, i.e. , into one learning paradigm. In applying the 
guidelines that were a part of the explanation, the individual 
materials preparer was given room for exercising his best 
judgment and for being creative in ' app?i.ying the paradigm in 
producing materials. 

The conditions of learning, then, were translated into 
learning paradigms for types of learning. These paradigms were 



3 Ibid., p. 164. 
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not given to the materials developers.^ Rather/ the paradigms 

were used to prepare examples of learning materials that embody 

the paradigms,, and these examples were given. Three materials 

samples were prepared for the use of the materials preparers. 

Sample A concerned higher-order principle learning (Type 8- 

Problem Solving and Type 7 - Principle Learning); Sample B 

concerned Concept Learning (Type 6); and Sample C concerned 

multiple-discrimination learning (Type 5) . Materials developers 

were informed that Sample A was to be generally followed for 

* 

al3. objectives labeled "A," Sample B for all those labeled "B/ 11 
and Sample C for all those labeled "C." 

Translating the learning paradigms to materials samples/ 
however/ required a number of decisions on treatment variables. 
Decisions had to 'be made concerning: 

. Use of "Organizers". 

. Step Size 
. Relevant Practice 
. Knowledge of Results 
. Sequencing within segments 
. Formatting 
. Pacing 

^Phrases such as "learning paradigms" were rarely used 
in comraurii cation wi th materials prepared or with anyone who 
was not a part pf the project team. 
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The learning paradigms {based on Gagne /, s descriptions 

of conditions of learning) implied the use of "organizers " to 

/ 

focus attention, indicate relevance, provide perspective, give 

a "roadmap" of the planned learning path. — all directed toward 

facilitating transfer and o.fcherwise increasing the efficiency 

of learning. ® There is evidence that the use of organizers 

£ 

can markedly increase learning efficiency. Organizers were 
used in introducing the learning, during the learning, and 
following the learning. 

* 

The decisions to be made on "step size" were considered 

crucial to the ultimate appropriateness and acceptability of 

the materials. Briggs points out that there is no common 

understanding of what "step size" means : 

There is no standard definition of "size of step." 

In programed instruction it has referred variously to 
(a) how difficult a response is to make, (b) how large a 
reading segment is presented before a response is required, 
(c) how much progress toward the goal is represented by 
one frame, { cl j how long it takes the learner to make a 
response, (e) whether or not the student responds correctly, 
and (f) how frequently reinforcement occurs. 7 



® Ibid. 

6 See the review by Popham, "Curriculum Materials," 323; 
also see Arden Grotelueschcn, "Effects of Differentially 
Structural Introductory Materials and Learning Tasks on 
Learning and Transfer, " American Educational Research Journal , 
V (19 68) , 191-202. 
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^Leslie J. Bricgs, "Learner Variables ( - 
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In the MMCD Project, "step size" was defined as the 
amount of learning required by the student to proceed from 
achieving one behavioral objective to achieving the next 
behavioral objective in the same learning sequence. Thus, 

"step size" was related to measurable behavior change. As 
indicated below in the discussion of pacing, the students 
were nevertheless individually given much control over the 
"step size" they actually used. 

Relevant practice was imbedded in the learning material 
for every objective, whether an intermediate behavioral objec- 
tive or a terminal behavioral objective. Moreover, self- tests 
were strategically Inserted in learning sequences, which had 
the dual benefit of giving the student feedback on his perfor- 
mance and letting him further practice the correct performance. 
About three or four practice situations (usually problems to 
solve } ware included in a ten-minute learning sequence for" a 
behavioral objective. The next behavioral objective in the 
hierarchical learning sequence would, in its instructional 
material, build on the just-learned behavior, often providing 
additional practice of the previous behavior or otherwise 
reinforcing that behavior. At the end of an instructional 
segment (about fifty minutes average learning time, but with 
great individual variation in time) , the self-test provided 
additional practice of desired behavior taught in the segment. 
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Knowledge of results, it was concluded, should be 
provided as soon as possible. A student response board was, 
therefore, among the media selected. This device is used by 
the student during the majority of instructional sequences 
(when the medium is not one or more humans (such as in counseling 
^sessions or seminars). The board gives him immediate feedback 
on his practice problems and on all self~tests (as he is taking 
each test). In the event a student's response is incorrect 

with regard to a given problem or tost item, he is expected 

* 

to follow the suggested learning prescription associated with 
the incorrect response. 

Sequencing within individual segments did not involve 
difficult decisions. A typical instructional segment might 
include from six 'to ten behavioral objectives. Individual 
materials preparers prepared complete segments (and, in some 
cases , complete topics having a number of segments ) . If the 
relationships among the behavioral objectives in a segment 
allowed for some minor sequencing options, the materials 
preparer was generally given the freedom to select the option 
that most suited him and any creative ideas he had in mind. 

Formatting decisions were considered to be of very 
great importance. The “format" included the physical layout 
of the printed page, the tone and rate of the spoken word in 
an audio tape, the nature of a computer printout, and a host 
of general considerations regarding these and other media. 
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It was believed that a "pleasing" image was important for the 
materials, that its absence would have a negative effect on 
the learning. It was not concluded that materials had to be 
elaborate. 

Since one of the major objectives, of course^ was to 
individualize instruction, it was decided that the student 
should largely conti'ol his own learning pace, provided he is 
progressing at the minimum acceptable pace. In order to 
facilitate such self-pacing, the learning materials were to be 
prepared so that a student could use them at different rates 
and amounts of interaction. (It was felt that a student should 
be able to use the materials as best suited his own "learning 
style.) Accordingly, it was decided t.o have a "high speed" 
path through the materials, plus a slower, highly structured 
path.. Also, it was decided to provide for multiple entry points 
in the material and to allow the student a degree of control 
over the sequencing of the . instructional events he participated 
in. 



Materials Specifications 



It was concluded that the only way in which the 
materials preparation schedule could be met was through a 
massive "parallel" production of virtually the entire 100 to 
200 hours of the course. The schedule allowed only three 
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months for developing the presentation-feedback strategies, 
generating the detailed materials specif i cations, producing 

/ 

the materials, and trying out and revising the materials until 
they achieved a satisfactory prevalidation quality. 

The present project staff alone was unable to carry 
out such a major undertaking, so it was decided that outside 
resources would have to be utilised. An assumption was made 
. that only experienced college instructors in economics would 
have sufficient knowledge and judgment to be able to prepare 
learning materials that met the specifications that would be 
provided. It was decided (1) that such economist-writers 
should be sought to supplement similar economist-writers 
already on the project, (2) a materials specifications "kxt ! ' 
should be prepared for each part of the course and used by , 
the economist-writers, and (3) a training program should be 
developed for all of the economist-writers , to train them in 
howto prepare materials to meet the specifications. 

The decisions regarding presentation strategies were 
largely documented in the form of materials samples. However, 
certain explanatory materials and guidelines were also 
provided for the use of selected materials developers . The 
materials samples, explanatory materials, and guidelines 
were put together in a "kit” for the economist-writers. Kits 
were essentially alike; -however, each kit also contained 
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specifications for a unique portion of the course. Thus , each 
kit was "tailored" for the economis t*-wr j.ter in terms of the 
part or parts of the course he was preparing. 

A kit contained the following items;® 

1. Economics Course Outline 

2. Behavioral objectives and sample test items for 
each segment for which the economist-writer was 
to prepare materials. 

3. One or more materials samples r including an 
example of a completed self-instructional segment. 

4 . A list of the summer i zed behavioral objectives 
for the entire course. (This helped the economist 
understand how a given topic fit into the course 
structure. It was also helpful in preventing 
omissions and redundancies of material.) 

5 . A statement of guidelines for economis t-wri ters . 
These were general guidelines for the self- 
instructional printed materials. 

6. Instructional exercise answer patterns. Economists 
received copies of these so that they could 
provide for various self- testing and controlled 
testing exercises in their materials, using a 
student response board. 

7. "Notes to Economists.” These were the notes 
economis t-wri ters received as an aid to them in 
developing materials. Many of these notes called 
for the development of subordinate behaviors in 
connection with learning to achieve a higher- 
level behavioral objective . 



listed . 



® See 



Appendix C 



for samples of some of the items 
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C. "Pinal Cliecklist: Economics Course Materials. 

The economist-writer was asked to use this check- 
list to make sure ho followed the guidelines 
given to him. 

Economics instructors from a number of universities 
I were invited to participate as writers. A one-day orientation 
| program was provided for those instructors who indicated an 
interest in participating^ At this orientation, the philosophy 
of the course was discussed and materials development specifi- 
cations reviewed. An outline of the entire course (see 
Appendix D ) was given to each participant at the outset. This 
was supplemented by summary listing of all of the objectives 
(cognitive domain) of: the course, broken down by topic and by 
segment within topic (see Appendix E) . Thus, if an economist 
felt his forte was in a particular topic, he could quickly 
review the summarized objectives for that topic and decide j.f 
the topic had more than visual' interest to him. (Normally, 
if an economist selected a topic, he prepared materials for 
all of the segments constituting i t , but there were some 
exceptions in the case of some very large topics.) 

The economist-writers left 'the orientation program 
with tentative writing assignments. A few decided that they 
did not want to participate after further reviewing the infor- 
mation they were given. Others immediately began their 
assignments. 
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Producing the Materials 



/' About twenty economists from a number of universities 
participated in the materials specification and preparation 
process. Some Worked on a full-time basis while others worked 
part time. The economics faculty and technology consultants 
were from the following universities: 

American University 
George Washington University 
Harvard University 
Nova University 

The Catholic University of America 
University of Maryland 
University of Pittsburgh 

f 

University of Toronto 

One of the functions of the economist-writer was 

to review the performance objectives and test items he received 

for accuracy and content validity. Materials released by the 

economist-writer were given the following treatment: 

A behavioral check, by a behavioral specialist 
using a checklist 

2. Preliminary editing 

3. A final instructional check by an economics 
professor v;ith extensive recent experience 
teaching introductory economics 
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4. Pinal editing by a professional editor 

5. Printing and duplication of packages for in-house 

i preliminary tryouts 

Part of the training given to the economist-writers 
was in the form of a critique of the manuscripts they sub- 
mitted. Perhaps one-half of the economists who began to prepare 
materials for a topic withdrew on their own accord or were 
asked to withdraw. In nearly all of these cases, certain 
qualitative criteria were not being met by the materials 
developer. Typically, the person was a poor writer or he 
found writing to be a very laborious task. 

The preliminary editing of the work received from the 
economists varied from a near rewrite to relatively minor 
changes. Good initial writing on the part of the economists 
could save substantial costs for the time and manpower required 
to revise it. The good, fast writers far outperformed their 
colleagues. In terms of expenditure per "unit" of material, 
one good writer might cost only one-third as much as an 
"average" writer and, in addition, meet due dates. 

Preliminary Tryouts 

The last step of the materials preparation task was 
to conduct the preliminary tryouts and to revise the materials 
based on the results. Collage students who generally resembled 
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the students at the Naval Academy in respect to academic 
background and achievement were used for preliminary testing 
of the material. These students "took" course materials in 
the same sequence as the midshipmen were to take them, but 
lone to two months prior to. when they would be given to the 
^midshipmen. ; 

The main purpose of these tryouts was to discover 
ambiguities, information gaps, terminology problems, prereq- 
uisite inadequacies, or any other problem that may hinder 

* 

achievement of the performance objectives. Materials that 
did not produce the desired learning were revised until they 
did. An interview technique was used in conjunction with item 
analysis and other techniques to identify where students 
encountered problems in the learning process. It was necessary 
to go -through several tryout-revision cycles in some cases 
before an instructional segment was satisfactory. (Appendix F , 
an excerpt from a project technical report, describes the 
procedures used for the preliminary tryouts.) 

The way in which learning packages were constructed 
made it possible to identify with great precision where a 
change had to be made in a segment to improve it. In the 
Self-Instructional Printed Packages, for example, an analysis 
of performance on the criterion test could- indicate the 
segment , page wi thin the segment , paragraph on the page , and 
sentence in the paragraph that caused the learning difficulty. 
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Comments on Parallel Development 

I The materials preparation process described above 

proved to be virtually trouble-free. Even though two 

semesters worth of mostly self-instructional course materials 

had to be produced in four months, this time span proved to 

be comfortable. Due dates were given to each writer that took 

into account the possibility that (a) he would meet the due 

date but his work would be of little if any value ox" (b) he 

would miss the due date by several weeks. Sufficient time was 

* 

always reserved to either (a) reassign the work to an 
economist-writer who always produced acceptable work or 
(b) completely redo the material using core members of the 
project team. 

While the materials were prepared in parallel, they 
were used in a linear sequence in the preliminary tryouts. 

In the MMCD Project, materials that needed to be ready soon 
for such tryouts always had the highest priority internally 
for needed edits, rewrites, reviews, typing, and duplicating. 
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CHAPTER IX 



COURSE EVALUATION 



It may be argued that informal evaluation defeats the 
purposes of evaluation because it yields data which are of 
dubious value for decision-making purposes . Hastings describes 
the purposes of evaluation as the collection of information — 

1. To be used as feedback to innovators for fxirther 
revision of materials and methods 

2. To provide information as input for decision- 
making by the schools about the adootion of 
course -content improvement packages*'- 

A definition of "evaluation" that resembles Ha sting's 
description is offered by Alkin: 

Evaluation is the process of ascertaining the decision 
areas of concern , selecting appropriate information r and 
collecting and analyzing information in order to report 
summary data useful to decision-makers in selecting among 
alternatives.^ 

In reviewing the literature in curriculum evaluation, 
Baker points out that as a subject matter "it lias not yet been 



lj. Thomas Hastings, "Curriculxim Evaluation: The Why 

of the Outcomes," Journal of Educational Measurement , II, Ho. 1 
(Spring, 19 66) , 27-28. 

2fJarvin C. Alkin, "Evaluation Theory Development," 
Evaluation Comment, II, No, 1 (1969), 2. 
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defined and bounded. He stresses the "considerable 

divergence" on many aspects of evaluation but points to an 

important area of agreement: 

Establishing an empirical basis for the revision and 
refinement of facilities, materials and methods appears 
to be a common objective of evaluation models . 4 

Stuff lebeam proposed a CIPP (Context, Input, Process, 
Product) model for evaluation purposes, that helps to imple- 
ment this type of approach.^ Similarly, Provus described a 

"discrepancy model" that is directed toward decision-making, 

* 

outlining five stages: (1) definition, (2) installation, 

(3) process, (4) product, and (5) cost-benefit analysis. ^ 
Stuff lebeam emphasizes that formal evaluations are 
typically poorly donej and lie suggests a number of reasons win 



^Robert I. . Raker, "Curriculum Evaluation," Review of 
Educational Research , XXXIX, No. 3 (1369), 339. 

4 Ibid. , p. 347. 

^Daniel L. Stuff lebeam, "The Use and Abuse of Evalua- 
tions in Title III and a Description of a Proposed CIPP 
(Context, Input, Process, Product) Model for Evaluation of 
Title III Projects." An address delivered at the Rational 
Seminar on Innovations sponsored by the Kettering Foundation 
and the U.S. Office of Education, Honolulu, Hawaii, July 1967 

6m. Provus, "Evaluation of Ongoing Programs in the 
Public School System," Ed ucat io n a 1 Eval u ation : M e w Ro .1 c s , 

New Means , ed. by R. W. Tyler, Sixty-eighth Yearbook of the 
National Society for the Study of Education, Part II (Chicago 
NSEE, 1969), pp. 242-283. 
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. educators dislike evaluation 
. they are untrained in it 
. it has not been done in the past 
. good guidelines are lacking 
. evaluation results have not been used 7 

| In a comparable vein, Stake criticizes the way in 

jwhich formal evaluations are being done, pointinq out that 

! 

they tend to be characterized by one or more of the following: 
. checklists 

. structured visitation by peers 
. controlled comparisons 
. standard i zed testing of students 3 

The evaluation task in the Multi-Media Curriculum 
Development Project was both planned and implemented within 
the framework recommended by the above researchers . In other 
words, an empirically-based decision-oriented process was 
employed. The evaluation task was a major undertaking, second 
only to establishing behavioral objectives in terms of man- 
power utilization . It was also the longest task , involving 
about a year and one-half to perform. While it presented 
problems, they were primarily a natter of reasonable profes- 
sional disagreements which were resolvable in an acceptable 
manner and within an acceptable amount of time. 



7 Stuf flebeam, "The Use and Abuse of Evaluations in 
Title III and Description of Proposed CIPP Model for Evaluation 
of Title III Projects," p. 4 . 

^Robert E. Stake, ".The Countenance of Educational 
Evaluation," Teachers College Rec ord, LXVJ.il, Mo. 7 (1967), 3. . 
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It should bo mentioned that the results of the evalua- 
tion process had major implications for all aspects of project 
performance. A major function of evaluation was to determine 
whether or not the instructional process was producing 
learning that met established criterion performance levels. 

The instructional process V7as to be revised when a perfor- 
mance criterion was not met. See Figure 12 for an illustration 
of the evaluation concept employed. 
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Fig. 12. — Illustration of Evaluation Concept Employed 
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Two-Semester Process 

; 'The preliminary tryouts preceded the validation trial. 
Essentially the same tryout-revision process employed for the 
preliminary tryouts was used in the validation trial. This 
process is described in Appendix g ("Excerpt from TR-5,'35: 
Preliminary Validation Report for Multi -Media Economics 
Course"). A brief explanation of how empirical data were used 
in deciding what revisions should be made is contained in 
Appendix H ("Excerpt from I.M. S87-R: Explanation of Materials 

Revision Process Rased on Empirical Results"). 

It was believed that a course could not be "evaluated" 
until it was "validated." "Validation " was directed toward 
insuring that the instructional process produced the intended 
learning outcomes, The validation trial was a "complete 
system test," while the preliminary tryouts were in effect 
"subsystem tests." It does not. necessarily follow that if 
each subsystem performs satisfactorily independent of the 
complete system that it will function satisfactorily as a 
system component. Thus, a function of the validation trial 
was to assess the extent to which the various components 
(segments, topics, pre- and post-tests, counseling, seminars, 
media, etc.) were integrated into an efficiently functioning 
whole. 
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Assuming that the integration is satisfactory, the 
various course segments should produce roughly the same 
achievement levels in the validation trial as they -did in the 
preliminary tryouts . Xf criterion performance levels are not 
( ach.ieved in the validation trial, a decision must be made as to 

t 

whether the rise in performance needed to meet criterion 
performance is worth the cost of revising the course, given 
the subjective probability that the course deficiency or defic- 
iencies have been identified. Many variables, of course, 
affect student performance, and it is often difficult to 
ascertain with precision the reasons why student performance 
is- below expectations. 

"Evaluation," on the other hand, was directed toward 
the question of how well a validated instructional system, 
operates within a given administrative-operational environ- 
ment. It was thought that a "validated" course could be 
disseminated to other educational institutions, where it 
would probably be subjected to an "evaluation." A course 

could be validated but fail the evaluation at the institution 
for any number of reasons, such as incompatibility with 
administrative policies and procedures, significant differences 
in the comparable target populations, or different attitudes 
on the part of the faculty. Accordingly, the process of 
evaluation, as viewed in the MM CD Project, was aimed at 
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determining how well a validated course "fits" at a given 
institution. If the fit is not satisfactory, a decision must 
be made regarding what changes might be made to produce a 

I 

i good fit and if the costs of making such changes are acceptable. 

f 

} 

It was therefore concluded that there should be a 
"validation semester" and an "evaluation semester" for the 
new course. The validation semester, it was planned, would 
be highly controlled, in the sense that all course options 
would be tested , de tai led responses would be captured , and 
great care would, be exercised to ensure that .if any students 
did not do well in the course it could be determined, within 
reasonable limits , if the instructional program were at fault. 

As in preliminary testing, it was always assumed that the 
instructional program was at fault if a student encountered 
learning difficulty, unless specific information indicated 
that there was some other cause. (Such information V7as 
sometimes obtained in interviews with students.) 

A Methodological Issue 

During the validation semester, students were not 
allowed to proceed through the course faster than a specified 
rate. This was done as a matter of simplifying course admini- 
stration and minimising administrative-operational problems 
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while materials were being tested. As a result , pre- and 
post-testing and other events were scheduled for certain dates 
throughout the semester. A student could not take a test or 
participate in a special event prior to the first date for 
which it was scheduled. Thus, while there were major differ- 
ences in time spent studying (see page 16 of Appendix iJ>no 
student could complete the course before another student. 

In the evaluation semester, on the other hand, it was 
planned that the course would be presented in its "fully 
operational mode. " And this was , in fact, accomplished. 

However , in connection with doing the planning for this semes- 
ter, a problem arose with regard to the evaluation methodology. 
Consultants to the Naval Academy recommended a form of the 

classical experimental design, iri which there would be a 
control group and an ’experimental group. A technical report 
from ETC argued the merits of using a clPP-type evaluation 
approach, as indicated by this excerpt from it: 
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In the fully operational economic analysis course, 
as it is currently designed, students will be able to 
proceed through the course at their own pace, make 
many of their own decisions on media usage and 1 optional 
enrichment areas, and otherwise exert a significant 
degree of control over when and how they study. In other 
words, the course is highly individualized in its intended 
operational state. It is very desirable that the course 
be evaluated in that highly individualized state. 

A methodological problem can arise if one wishes to 
perform controlled media research during the "evaluation 
semester." An experimental design necessarily calls for 
extensive control of variables. And only when an experi- 
ment is carefully controlled are its results of value. 

The type of control generally required, however, is 
directly at odds with the concept of a highly individual- 
ized course. The very essence of a highly individualized 
course— the flexibility and options — presents the problem. 
It is possible to be faced with the alternative of either 
conducting a carefully controlled experiment or evaluating 
a highly individualized course of instruction . 

In the present case, it is of critical importance that 
the evaluation apply to the course as it will be operating 
in the future. Any media studies conducted during the 
evaluation semester should disturb as little as possible 
the intended course operations. Only then will the results 
of the evaluation be useful. And of course, the results of 
media studies which do not disturb the normal operations 
will have greater value for a curriculum development model 
than studies conducted in a temporary artificial atmosphere. 

One last point should be made regarding evaluation 
methodology. There is usually a "shakedown" period after 
a new system is first placed into full operation. During 
this period, it may become neccr.sa.ry to modify the system 
slightly to overcome an unexpected problem. The previous 
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tryouts of the economics course have tended to validate 
its subsystems and its plan of operation. The shakedown 
period for the operational course will be the evaluation 
semester/ which begins in September 1969. The evaluation 
/ methodology must be such as to permit necessary changes 
to be made in the administration of the course .during 
■this period. 

Evaluation activities may be described as falling 
into four basic categories: 

1. Student performance 

2. Student and ins true tor acceptance 

3. Course management 

4. General operational environment 

The ultimate measure of the success of a learning 
program is, of course, the performance of the students 
in achieving the learning objectives. 'However, the 
other three categories of activities listed above are 
important in facilitating or hampering the achievement 
of the desired performance. Each of the four categories 
is discussed below in terms of objectives, data required, 
method of data collection, and data analysis. 9 

The Academy and its consultants accepted and modified 
CIPP evaluation approach that was proposed. 



Validation Trial 



The validation trial was conducted over a semester's 
time, as planned. The following classes of data were collected: 



. Learning time 

. Learning -imbedded student responses 
. Self-test student responses 



‘^Educational Technology Center, TR-5. 37. 
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. Criterion post-test responses 

* Medium-use record per student 

* Student course critique information 

* Instructor course critique information 

I * Critical incidents 

These data were analyzed and used as the bases for 
course revisions. Appendix G ("Excerpt from TR-5.39: Final 
Validation Report") describes in detail the validation 
activities and the results. 

As stated earlier, the validation trial assumed that 
the quality of the instructional segments had been confirmed 
through the preliminary tryout-revision process. In that 
process, some instructional segments went through as many as 
five empirically— based revisions , though this was infrequent 
(two or three were typical) . It was important that the material 
be virtually "pre-validated" for use in the validation trial 
because only one empirically-based revision could result from 
that trial. This was the case because all of the subjects 
took the course at one time. A revision oould not be 
"re-tested" in the validation semester because there were no 
more subjects available. 

10 New subjects could not be easily brought in, since 
it might take a new subject a month or more to acquire the 
entry behavior necessary to test a revised segment located 
in the middle of the course. 
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Since any revisions done following the validation 
trial could not be properly tested before the evaluation 
trial, extreme caution was exercised to prevent "over- 
revising." Over-revision was viewed as making changes in 
"problem" material beyond those clearly indicated by. empirical 
data. If material were modified without a clear-cut empirical 
justification, it would be possible to make the material less 
effective than it was prior to this revision. Hence, tight 
controls were employed to review revision rationales before 
the material could leave the revision process. 

Z^lso, the standardization of the decision-making 
rules as pertain to revisions \?as considered a requirement. 

Decision-making guidelines v;ere promulgated toward this end 

11 

and review points’ in the revision process were established. 



Evaluation Semester 



Evaluation activities fell into five areas: 

1. Student Performance 

2. Student and Instructor Acceptance 

3. Course . Management 

4 . Course Environment 

5. Special Media Analyses 




“An 

found in Apoendi; 

of Materials Revision rivuLBs uaoeu on i^»uiua ,, 

where a step-by-step decision-making procedure is described. 
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S tud e nt Per f o rmance Ev aluation 



The primary purpose of this area of evaluation was to 
determine the efficiency of instruction. Efficiency was 
measured in terms of learning-time-taken and achievement 
scores on criterion-referenced examinations. Two classes of 
data were required: 

1. Achievement scores on criterion referenced 
examinations/ for each student on each control 
test 

2. Amount of learning time taken by each student 
on each instructional segment. 

The criterion-referenced control tests were admin- 
istered to students by the course administrator/ in an area 
provided for such testing, A student was permitted to take 
such a tost as soon as he believed he could pass it. If he 
failed to achieve criterion performance in two tries/ he was 
required to see his professor/ who would make a diagnosis 
and prepare an individual learning prescription. (This 
arrangement was intended to deter students from taking 
criterion tests more than once/ without having studied/ in 
the hopes they might pass some by chance. It was also 
intended to help identify/ very early/ the students who may 
have a serious . learning problem which needed the diagnostic 
skills of the instructor.) The final achievement score 

12 Based on experience/ the probability of this 
happening v/as less than two per . cent. 
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obtained by a student on a control test was used as the 
measure of his "performance." 

Time-taken was recorded on a segment-by-segment basis 
by each student. The student submitted his study time record 
I to the Course Administrator at the time he took a control 
Itest. The total study time accumulated prior to the student's 
passing the control test constituted his learning time-taken 
for the level of achievement he obtained as indicated by his 
test score. 

Student-Instructor Acceptance Evaluation 

It was thought to be very important to take into 
account the attitudes and opinions of the students and the 
instructors involved in the multi-media course. Even if the 
course were effective from an instructional or learning stand- 
point, it was also important that the course be "acceptable” to 
those who are involved in it as instructors or students. There 
are different levels of acceptance, ranging upward from barely 
tolerable. The key level of acceptability, it was thought, 
was the level at which a student or instructor would be willing 
to continue with a course such as this in the next semester. 

The purpose of this evaluation area (a) to ascer- 

tain if students and/or instructors were willing to continue 
in a course designed as this one, and (b) to determine what 
aspects of tlie course might be changed to increase the accept- 
ability while meeting the learning objectives. 
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Students' and instructors' comments on specific 
aspects of the course, as well as the course in general, were 
routinely solicited. The specific aspects of the course on 
which comments were solicited included each media-materials 
combinations and other planned instructional activities. 

Critique sheets were issued with both instructor 
lifici fcGiri 1 g s fcu don't jn£i t g 3rd. ex 1 s sncl o tVi c sg tc \ ex cIg 

available to the instructors and students. These sheets were 

sufficiently structured so that information was solicited 
regarding specific and general aspects of the course. How- 
ever, the reply requested was "open-ended," so that no 
restrictions were placed on obtaining possible negative 
comments . 

Critique sheets turned in for evaluation purposes were 
subjected to an analysis to determine (a) what aspects of the 
course were reported as best liked and (b) what aspects were 
negatively criticized. Absence of a negative response on an 
aspect of the course was taken to imply that that aspect was 
acceptable. The specific negative and positive comments 
served as the partial basis for revising the course admini- 
stration to increase the general level of acceptability . 
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Course Man eg ame nt 






The course management concerns instructor-student- 
media-materials logistics i schedules, learning prescriptions, 
and other aspects of providing for and monitoring instruc- 
tional activities. Course management planning was done by a 
course Administrator and 1 the instructors within the context 
of previously established course management policies and 
procedures. A Course Administrator was responsible for admin- 
istering the plan (he performed no instructional or formal 

guidance function) . 

The purpose of this evaluation area was to assess the 
adequacy of the course management plan and its administration 
and to ascertain what improvements might be made. Accordingly, 
information on problems that arose regarding course manage- 
ment was needed. This information, it was decided, should 
include a description of the problem, the date it first was 
detected, the circumstances under which it arose, what was 
done about it, who was involved, and what future action seemed 
advisable. The multi-media course instructors, the course 
administrator, and the liaison personnel of the contractor 



were each given a notebook and instructions for recording 
critical incidents. The data, thus recorded are currently 
being analysed to determine how unde aired events might be 
avoided in the future. 
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General Operational E/ivi ronment 

A highly individualized instructional program cannot 
be designed without regard to the constraints that* may be 
imposed by the general environment in which the program is to 
be carried out. Similarly, such a -course cannot be success- 
fully introduced into a conventional educational environment 
without the environment being changed to some extent. The 
"environment," as used here, refers to policies, procedures, 
staffing, facilities, and similar factors that constitute the 
situation within which the program, or course, operates. 

Several examples may be given of the kinds of 
"conflicts" between the old and new often created by intro- 
ducing such a course. A grading policy may have been estab- 
lished only after long and careful deliberation, but a 
question arises as to when "mid-term" grades should be sub- 
mitted if a program is self-paced and students are at many 
different places in the course. Similarly, a media-materials 
resource center has a set of new requirements to meet when it 
must serve a highly individualized, multi-media course. 

The purpose of this area of evaluation in the KMCD 
Project was to assess the effects of the course on its opera- 
tional environment and the effects of the environment on the 
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operation of the course. It was believed that such an 

assessment would f provide information regarding what changes 

in the course or its environment might be advisable ,to further 

* 

the learning goals. 

Information regarding conflicts between the multi -media 
course and its operational environment had to be recorded. The 
recorded information was to include a description of the con- 
flict, the date it was first detected, its implications, what 
was done about it, and what future action seemed advisable. 

The multi -media course instructors, the course administrator, 
and the contractor liaison personnel each were given a notebook 
and instructions for recording operational conflicts. 

Staffing, Organization, and Costs 
Some discussion of staffing, organization and general 
allocations of resources by phase may be of value at this point. 

As would be expected, the project was staffed with 
professionals having a broad grasp of educational technology, 
plus experience with innovative educational projects. Figure 
13 gives the organization chart for most of the project. 

Nearly every one of the sixteen persons indicated on the chart 
was involved in one or more advanced development projects prior 
to joining the MMCD Project. 
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Another characteristic of project personnel was that 

most of them were qualified to perform more than one function 

* * 

in the project. As a result, many of the staff took part in 

I 

I many tasks, still another characteristic that deserves 
I mention is that all but a few staff members had once been 
full-time teachers. 

The cost’s of any curriculum development effort are 
dependent on many things. Costs must be measured in terms of 
resources expended. Based on the 11MCD Project, the cost-ratios 
for the three phases of the project were about 2:2:1, respec- 
tively for phases I, II, and III. These cost ratios should 
be viewed as simply a frame of reference. 

In an extremely complex educational problem area, the 
first phase might require a higher proportion of the total 
resources than the 405 indicated by the above ratios. 

In an uncomplicated situation, the proportion required for 
the first phase might be less than 40 per cent. Similarly, 
the proportionate cost of Phase II can rise or fall based 
on the complexity and amount of the software that needs to be 
developed. . • 
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CHAPTER X 



CONCLUSIONS AND RECOMMENDATIONS 

It is appropriate that this report comment on the use- 
fulness of the development model for other course developers. 

Majof Conclusion 

The value of the development model that has been 
described in the previous chapters is suggested by the 
characteristics of the course that was produced using it. 

In other words/ the course is the test of the model. A 
summary of the Final Evaluation Report is given in Appendix J. 
Note that among the students who completed the two-semester 
course in six weeks were (a) the student who spent the least 
amount of time studying and (b) the student who spent the most 
amount of time studying. This demonstrates the flexibility of 
the course in terms of the learning styles of individual 
students. In this connection, note also that ninety-eight 
percent of the students achieved eighty percent or more of 
the core objectives and nearly all of them achieved sufficient 
enrichment objectives to raise their grades one or more levels. 
Further, student and instructor acceptance of the course each 
semester it has been presented has been virtually unanimous 
and without reservation. 
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A course which can guarantee learning while earning 
the acceptance of students and instructors alike is of con- 
siderable importance. This course has the additional features 
however, of (a) requiring one-third as many instructors as 
previously and (b) requiring one-third as much classroom usage 
as previously. Reductions in learning time, number of instruc 
tors required, and classroom usage can be translated into sub- 
stantial cost savings. 

Further, the entire course is replicable, and not 
dependent on non-transportable, high cost media, and its 
modular design and structuring options permit a user to 
tailor the course to his own circumstances. If necessary, the 
course could be presented in a completely self-instructional 
mode using only low-cost, battery-operated media that a child 
could carry. Thus, this validated course could be sent to 
places that have no economics instructors and no educational 
facilities*'*' These characteristics of the course can also be 
translated into substantial cost savings compared to con- 
ventional educational practices. 



In cases where there are qualified economics 
instructors available, the course should not be structured as 
completely self-instructional, but should use an instructor 
to conduct seminars , to work with students individually on 
special projects, to do counseling, and to perform a number of 
other valuable .functions . 



- 124 - 



O 



KDU< ' ATlONAf * TBt^ll^OljOCTV CEKTKK 



In view of the results, it must be concluded that the 
process by which the course was developed merits careful 
consideration by other course developers. 

Usefulness of the Model 

The model resulting from the MMCD Project provides 

a number of benefits to the educational community, but the 

* 

model is not without its limitations. Some of the advantages 
of the model will be discussed first, follov7ed by a discussion 
of certain apparent limitations. 

Advantages of the Model 

An important characteristic of the model is that it 
is not theoretical but empirically-derived. This gives the 
model much more authority than theoretical models, regardless 
of the elaborateness of their rationales. 

Another important aspect of the model is that its 
starting point was the "state of the art" of educational 
technology. The project was planned using the leading or 
most promising theoretical approaches to various aspects of 
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curriculum development and, in significant part, translated 

2 

that theory to technology. Thus, the model that was produced 
represents an advancement in educational technology. 

A third advantage of the model is that it provides 
. information about techniques which were successful indivi— 

i 

j dually and collectively. Some techniques may be successful 
, in isolation but unsuccessful when used in conjunction with 
certain other techniques, rationales, or approaches. In the 
model that was produced, such aspects are integrated, and 
there is no question of compatibility. 

A fourth advantage is that, in effect, the model 
suggests what not to do. The development model, as reported 
here, includes descriptions of approaches that did not work. 
While one should not conclude that an approach which failed 
in the MMCD Project will fail whenever and wherever it is 
tried in the future, it probably has a greater likelihood of 
failing than the approach found to be successful in the MMCD 
Project. 

2 What is a "leading" theory or a "most promising" 
approach is a matter of judgement. The Educational Technology 
Center judged that it used the leading or most promising 
theoretical approaches, based on a knowledge of the research 
in the field of educational technology, including the results 
of projects funded by the Office of Education's Bureau of 
Research. 
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Fifth, there is much flexibility "built into" the 
model. For example, it is possible for each of the three 
phases to be successfully conducted by an entirely, different 
staff. And, indeed, this may be very important because 
different skill mixes are needed by project personnel in the 
major phases. Accordingly, one organization could produce 
the specifications for a new course (phase I) , another 
organization could develop the instructional materials (phase 
II), a third organization could conduct the validation tryout 
(part 1 of phase III) , and any number of organizations could 
then evaluate the course at their own institutions (part 2 
of phase III). 

A sixth point in favor of the model concerns the 
philosophy that underlies it--namely, the systems thinking. 
Systems thinking is an attitude, just as optimizing among 

I 

alternatives is primarily an attitude. This kind of thinking 
can be applied by anyone concerned with planning and imple- 
menting education and training programs. The somewhat sophis- 
ticated modern management techniques that are part of the 
systems approach require special training or experience before 
one can hope to apply them successfully. 

Finally, the development model is useful as a starting 
point for others, as a frame-work for further improving the 
model. 
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Limitations 



The development model that emerged is by no means a 
panacea. While an important improvement over approaches that 
have preceded it, it is nonetheless capable of being signi- 
ficantly improved itself. Some of the more apparent ' limita- 
tions of the model are described below. 

First, it is clear that the model requires highly 
skilled professionals — behavioral scientists, test and measure- 
ment specialists, media specialists, systems engineers, etc. — 
who are always in short supply. Also, it -may well be the case 
that skill in using the model is as important as prior quali- 
fications, perhaps including knowledge about the model. As a 
result, it would seem that training programs to teach the 
pertinent knovl edges and skills required to us e the model might 

t 

be necessary before any extensive use of the model could result. 

Second, the management of the curriculum development 
process was obviously an important element in the evolution of 
the model and the success of the Multi-Media Economic Analysis 
Course produced with it. The management approach used in the 
MMCD Project is not uncommon in leading industrial organi- 
zations. However, modern management techniques have by and 
large not been applied in our state and local educational 
systems. As a result, there are relatively few "modern 
managers" available who know enough about education and 
educational technology to conduct projects such as the 
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MMCD Project. A serious question is raised of whether or not 
such managerial expertise can be found in adequate supply in 
the educaticnal community to manage major projects, that apply 
a development model similar to the one reported here. 

A third limitation is the amount of investment required 
to use the development model. There is no "minimum" cost to 
using it, but it seems clear that developing a course using 
this model is going to cost more than developing a course in 

t 

the conventional fashion. It should be understood, however, 
that only the initial costs may be higher. If the model is 
used to produce a replicable course that guarantees learning 
and this course is compared v;ith a conventional lecture course, 
the overall economic benefits of the replicable course that 
guarantees learning may far surpass those of a good lecture 
course. A replicable self-instructional course that guarantees 
learning could be used throughout the country. There is no way 
to replicate a live lecturer~~he can only be in one place at a 
time . 

The fourth limitation is that the model needs to be 
exercised under many different conditions, as a way of both further 
testing the model and strengthening it on the basis of empirical 
data. 
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Recommendations 



Empirical ly-derived curriculum development techniques 
were in existence prior to the derivation of the ‘curriculum 
/development model reported here. However, a search of the 

i - 

I literature yields valuable case histories, but no development 
model per se . Accordingly, the model reported here appears to 
have unique value because other empirically-derived models 
cannot be found. This reason alone recommends that the 
elements of the model, including techniques that 'worked and 
techniques that did not work, be considered by the person 
contemplating a curriculum development project. 

In view of the success of the Multi-Media Economic 
Analysis Course produced using the model, it is reasonable to 
recommend that the model be used in further curriculum 

development activities. This could have several advantages. 
First, it could produce courses that guarantee learning, 
which is a characteristic that few courses have. And such 
courses could be acceptable to the students, teachers, and 
administrators involved with them. In addition, further use of 
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the model could help to "test" 



the model in more varied 
situations. This would includa different subject 'matter, 
different student populations, different resources and con- 
straints, etc. The model itself could be revised based on such 
tryouts, so that it continues to be improved for future 
application. 

The model also could benefit from continuously incor- 
porating and translating to practice the results of new 
research on learning. For example, the ability to more 
effectively individualize instruction can be enhanced through 
further research on learning styles. Further research on 
creativity or in the broad area of the affective domain couid 

also benefit the model and thereby help to produce better 
curricula. 

I If the model were improved through contributions from 

a PPlications of the model in diverse situations and from 
learning research, its utility would be similarly improved. 
Presumably, it would provide, as a guide, many alternative 
strategies at the key decision points, plus the criteria to 
help the curriculum planner select the alternative that is 
most appropriate to the requirements of his situation. This 
could immensely strengthen the model as a decision-making aid. 
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However, with improvements, the model would become much 
more complex than it currently is. It is -therefore recommended 
that the model be converted to the form of a computer.-based 
^imulation. This would permit curriculum planners to "test" 
the probable effects of alternative decisions regarding their 

I 

planned curricula. Having the development model in the form 
of a computer-based simulation would facilitate the use of the 
model (since the computer can simultaneously manipulate many 
variables) but also Quickly facilitate modifying the model to 
incorporate new information. 



- 132 - 



O 



H1IUCATIONAT. I Kt-HNOI-OnY CliNTKH 



I 



APPENDIX A 



Description of Components of the Course 
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COURSE MATERIALS 



MULTI-MEDIA ECONOMIC ANALYSIS COURSE 



I The Multi-Media Economic Analysis Course developed by 
the Educational Technology Center of Sterling Institute 
consists of the following material: 

o Seven (7) volumes of self-instructional texts 

i 

I 

o Three (3) self-instructional audio-workbook packages 
o Eleven (11) computer-based simulation models, 
o Nine (9) 16mm films 

o Criterion-referenced pretests and post-tests for 
, use with the above instructional material 

o' Instructor's Guide/ Course Administrator's Guide, 
Student Handbooks 

' 1 1 * , 

Self- Instructional Texts 

The contents of the seven volumes of self-instructional 
texts are listed below. The approximate student study time 
required to master the material in each volume (i.e., to 
successfully pass the associated criterion tests) is also 
given. Note that the last of the seven volumes is called 
"Enrichment. " 



Approx. Study 



Volume 


Time (hours) 


Content 


I 


8-11 


Nature and Methods of Economics 
Production Possibilities 
Demand, Supply, Equilibrium 
The Circular Flow 


II 


10-16 


The National Income Accounts 
The Theory of Income Determination: Part I 
The Theory of Income Determination: Part II 
The Role of Fiscal Policy in Income 
Determination 


III 


8-11 


The Role of Money & Banking in the Economy: 

The Role of Money & Banking in the Economy: 

The Role of Money & Banking in the Economy :I 


IV 


7-12 


The Theory of Demand (Value) 

Costs of Production & Supply: The Short Run 
‘Costs of Production & Supply: The Long Run 
Industry Equilibrium in a Perfectly 
Competitive Market 



Volume 


Approx. Study 
Time (hours) 


Content 


V 


9-13 


Monopoly Markets 
Monopolistic Competition 
Oligopoly Markets 

The Theory of Factor Demand and Supply 


VI 


8-12 


Specialization and Trade 
International Trade 


Enrichment 

i 

f 

♦ » 


7-9 


A Spectrum of Economic Systems 
The Influence of Government and Foreign 
Trade on National Income 
The Product and Monetary Markets 



Self-Instructional Audio-Workbook Packages 

The Self- Instructional Audio-Workbook Packages serve as an 
alternate media-mix for parts of the first three volumes of Self- 
Instructional Texts. Each package contains both a workbook and 
one or more cassette-type audio tapes. The contents of the 
packages, and the approximate student study time, are; 



Audio- 

Workbook 


Approx. Study 
Time (hours) 


Content 


A 


1 H 


Demand , Supp ly , Equi 1 ibr ium 


B 


2H 


The Theory of income Determination; 


C 


3 


The Role of Money & Banking in the 




i 


Economy; III 



Computer-Based Simulation Models 

Eleven economic simulation models were tailored for use on 
a commercial time-shared computer system. The eleven simulations, 
all in macro economics, are listed below. The simulations may be 
used by the individual student or by groups of students (e.g., a 
demonstration before a group). A minimum of 35 minutes is recom- 
mended for exercising a simulation model. 

o The Investment Multiplier ' . 

o A Model With Induced Investment 
o Fiscal and Monetary Policy 
o Government Tax and Expenditure Policies 
o The Demand Model With a Money Market 
o Demand Model With a Foreign Sector 
o The Complete Demand Model 
o Full Employment Output 

o Capacity Output and the Production Function 
o Effective Demand and Potential Supply 
o Aggregate Demand and Potential Money Supply 



Motion Pictures 



Nine films were rented for uje during the course. These 
films were selected because of their compatibility with course 
objectives. It is planned that other audio-visual materials 
may be specially developed for use in the course, perhaps 
supplanting one or more of these films. Each film is about 
30 minutes in length. The titles of the films, all of which 
were from the Coleman series The American Economy , are: 

j o Introducing the Price System 

o Gross National Product £ Its Cousins: Part I 

o Gross National Product & its Cousins: Part n 

o How Money Expands and Contracts 
o Banks for Bankers: The Federal Reserve System 

o Money and the "Fed": Part I 

o The Case of one Firm 
o The Case of Few Firms: Part I 

o The Case of Few Firms: Part II 

Criterion-Referenced Tests 



Criterion-referenced pretests and post-tests are provided 
for all of the material mentioned above. 

■* 

Instructor's Guide, Course Administrator's Guide, Student Handbook 

These guides and handbook describe how the course is operated 
and what options exist for the student, the instructor, and the 
course administrator. These materials tell how the course may be 
operated in the fully self-paced mode. 

Use Options . 

The course may be operated with only Self-Instructional texts. 
The "minimum" course shcuid include Vol. I (Basic Concepts) and 
Vol. II (Macro Economics) or Vol. I and Vol. m (Micro Economics). 
Volumes I through VI, plus the Enrichment volume, constitute a 
very substantial course for one semester (and, in terms of weight, 
about 10-12 pounds of text). The Self-Instructional Audio Packages 
are, as mentioned earlier, alternate media to some of the self- 
instructional material appearing in Volumes I through hi. 

If one adds the simulations, the 16mm films (or their sub- 
stitutes) , and includes the recommended seminars and group discus- 
sions, the demands on the student's time similarly increase. The 
simulations, however, may challange the most advanced student in 
the course. Films are becoming available that will show economic 
processes while they are happening in real life, an important 
. aspect of communicating relevance. And the seminars represent 
critical opportunities to integrate and synthesize learning, to 
generalize to other situations, and employ group dynamics to 
otherwise stimulate the students. 
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Comments on Media Applications 



Self-Instructional Printed Packages with Student 
Response Boards . 

These packages combine large-step, linear 
programming and adjunctive programming. Specially-developed 
models served as guidelines for applying this hybrid program 
/ ming style in developing materials for particular objectives 

Great care was given to taking the student 
from the known to the unknown, in providing him with 
ample examples and illustrations, and in giving him 
sufficient practice of the desired performance. 
Learning-imbedded tests are used "in the learning" and 
at the end of each package there is a diagnostic self- 
tCst. 

The Self-Instructional printed Packages are 
used with a portable student response board, which 
can be used in the dormitory room, the library, the 
classroom — wherever the student studies best. Nearly 
all test items included in a package', including the 
learning-imbedded items and the diagnostic self-test 
items, are programmed for the board. The student 
responds to multiple choice items by selecting and 
depressing one of four response keys. If he selects 
the correct answer, the indicator advances to the 
next test item number. This "advance" informs the 
student that his answer was correct. If he selects 
a wrong answer, the key remains depressed and the 
indicator does not advance. The indicator does not 
advance until the correct answer is selected. In 
this way, the student' is always given immediate 
feedback on his response, whether incorrect or correct. 

The response board has 15 different answer 
pattern settings, including a special setting for 
"control" tests — that is, tests used for grading 
purposes. On the control test setting, the unit 
advances each time a key is pressed and the student 
gets no feedback during the test. An important 
feature of the student response board is that a com- 
puter processable punched card can be produced each 
time a test is taken. Such records are useful for 
validating learning materials as well as recording 
a student's relative progress. 

In summary, the Self-Instructional Printed 
Package with Student . Response Board was selected as 
the primary instructional medium in the course because 
its characteristics best matched the micro and macro- 
media selection criteria. 

POOR ORIGINAL COPY -BEST 
AVAILABLE AT TIME FUMED 




Self-Instructional Audio Packages 



A Self -Instructional Audio Package consists 
of cassette audio tape, a workbook, and a portable tape 
recorder. The student response board is used for the 
practice problems and self-tests. These packages 
employ the same hybrid programming style as the Self- 
Instructional Printed Packages, but with certain added 
features. .These packages were programmed following 
audio-practice-audio-practice cycles. The idea was to 
deliver the audio in a crisp fashion, using a "large 
step si.ze," unimpeded audio flow. No audio is given, 
during intervening practice and problem-solving periods 

The Self-instructional Audio Packages were 
selected for learning situations requiring explana- 
tions of complex graphs , where the audio can be used 
to draw attention to and explain elements and re- 
lationships portrayed by such graphs. It was also 
assumed that some students would prefer audio-oriented 
instruction or be able to learn from it easier than 
from visually-oriented instruction . 

* * 

Computer Simulations 

An Economics Simulation Model consists of a 
computer program which the student can exercise at a 
computer terminal, a workbook which contains informal 
tion about the model, and a cassette audio tape which 
explains in detail its workings and its relationship 
to other parts of the course. There are approximately 
twelve such models.. 

The rationale for using simulation was to 
provide for cognitive structure, s that transcend, 
integrate, unify, and reinforce the learning of groups 
of related instructional topics and course segments; 
They are easily able to provide highly complex problem- 
solving activities dealing with variables from many 
different parts of the course. 

16mm Film 



These black and white films are approximately 
one-half . hour in length each and were selected from 
Coleman's "The American Economy" series. While motion 
pictures represent an extremely high-cost medium if 
original films are produced, the opportunity to rent 
or purchase applicable films renders this medium 
feasible for possible use on a relatively extensive 
basis. Films were selected that generally teach the 
same student performance called for in some segments 
of the course. The films represent a different pro- 
gramming approach. 
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Seminars 



There are simulation seminars , remedial semi- 
nars, and general or enrichment seminars. The role of 
the instructor is to serve as a seminar leader. A 
seminar may involve from 2 or 3 students to 15 or 20 
students. Seminars were selected for a number of 
reasons, such as to provide a means for sharing 
insights and experiences, to enhance group identifi- 
cation, and therefore to increase or maintain student 
interest and motivation. 

Counselin g 

Counseling is done by the instructor, usually 
(not always) in a one-to-one student-instructor 
relationship, and preferably with some degree of pri- 
vacy. Counseling sessions are viewed as critical 
opportunities for the instructor to motivate students 
and to relate activities to their interests and goals. 
Counseling was selected as the only currently avail- 
able means of diagnosing the more difficult learning 
problems and determining the enrichment learning needs 
of the very advanced student. 

Lecture-Discussion 



In the lecture-discussion situation, the 
instructor first presents information, using audio- 
visual aids as appropriate. A brief group discussion 
follows each Jcey pcint presented. The lecture- 
discussion was selected primarily as a means of 
enabling the instructor t6 stimulate, motivate, or 
otherwise cause enthusiasm over the course. A secon- 
dary reason for selection was to inform students of 
what is to be expected in the course. 

Tutor 



In tutoring, the instructor, or an advanced 
student selected by the instructor, meets individually 
with a student having learning difficulties. The 
individual tutoring is aimed at well defined trouble 
spots uncovered by tests . This is a very costly 
medium selected only for use in last-resort remedial 
situations, after a student has unsuccessfully used 
other remedial media. 
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Related Texts 



The two best known introductory economics 
texts are used as "related texts." The instructor 
is free to designate other materials as similarly 
"related.” Related texts were selected as readily 
available media for remedial purposes or enrichment 
purposes, depending on the specific learning pre- 
scription. 



C omputer-Based Test Analysis 

All pretests and post tests for core and 
remedial segments are automatically recorded on 
computer-processable punched cards, as a result of 
the use of the student response-feedback boards. 

These cards are pre-punched with each student's 
identification number. 

! This approach v/as selected as an efficient 

means of obtaining student performance data that can 
be manipulated bv a computer for analytical and record 
keeping purposes. It allows the student to take a 
tost at virtually any time, without an instructor 
being present. It also avoids problems of (1) tying 
up expensive equipmen t during student "think" times, 

(2) creating Student queues at computer terminals, 
and (3) general problems of logistics so often en- 
countered in testing. 
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Sample of Items from Kit for Materials Preparers 



The sample includes the following: 

#1. Materials Preparation Course Outline 

This is the outline of the course given 
to the materials developers. 

#2 . Final Check List : Economics Course Materials 

The economist-writer used this checklist to 
make sure he. had followed the guidelines given 
to him. 

#3. Guidelines for Economist-Writers 

These were general guidelines for use in developing 
the S~I packages. Note that the type of learning 
for each objective is indicated in this document. 

#4. Notes to Economists 

These are examples of the types of notes economist 
writers received as an aid to them in developing 
materials. Many of these notes call for the 
development of subordinate behaviors for a be- 
havioral objective. 

#5. Example of Detailed Behavioral Objectives, Test 
Items, and Hierarchy Charts in the Form They Are 
Given to Materials Developers. 

This example concerns the topic "Specialization," 
which consists of two segments (segments-# 6 and 
An attempt was always made to assign all of the 
segments in a given topic to the same materials 
developer. 






MATERIALS PREPARATION 
Course Outline 



I Introduction 

A. Purpose of Course 

To prepare you to prepare learner-centered 
instructional materials in accordance with 
a format specified for this project. 

- ?• ■ . 

B. Scope of Course 

1. The nature, scope, objectives and status 
of the project. 

2. The technical concept of individualized 
instruction . 

3. Operational concept for individualized 
instruct ion . 

4. The model learning package; the design 

of the model learning package; the rationale. 

5. Given "inputs," prepare "outputs," review 
and analyze. 



II The Nature, Scope, Objectives & Status of the Project 
A. ESOE/NA - Multi Media Course Development 

1. ModOl Development (OE) 

2. Fixed Price/Schedule 

3. Product Econ. Course (HA) 

4. Behaviorally Oriented - 3riggs/Ga.gpe model 
' 5. Status 

a) completed (show materials and hierarchy) 

b) yet to do 

Phase II 
- Phase III 



III Technical Concept 

A. Behavioral Hierarchy 

B. Individualized Instruction 

C. Validated Materials 

D. Gagno's Model 

* E. Pitfalls and Misconceptions 



POOR ORIGINAL COPY -BEST 



Operational Concept 



A. Remedial , Core , Enrichment 

B. ‘ Dymedia 

C. Multi-media 



Format for Materials 

A. Cover Page 

. Responsibility: Sterling/Tripp 

. Contents: See Illustration A 

B. Table of Contents 

. Responsibility: Sterling/Tripp 

. Contents: See Illustration B 

C. General Instructions 

. Responsibility : Sterling/Tripp 

. Contents: Prerequisite Segments 

Dymedia Settings 
Package /Topic information 
Use of Segment/Dymedia, etc. 

(boilerplate mechanics) 

D. Introduction to Topic 

. Responsibility: Economists 

. Contents: Brief Narrative (2-3 double spaced 

pages) stating in very direct terms 
(1) the overall terminal objective of the topic 
(?) the significance of this topic; why it is 

important, how it relates to other topics, etc. 

(3) how the topic presentation is organized, what 
the student will learn first (Segment x) , 
what he will learn next (Segment x +1), etc. 

(4) pitfalls and/or difficulties (false friends!) 
which the student can expect to encounter. 

Note: The purpose of this topic introductory 

' statement is to 

(1) motivate the learner 

(2) provide the learner with an "organizational 
framework" and/or a perspective for learning. 
Tell him what he is expected to learn and why; 
alert him to problems of semantics, etc. 

E. Segment Cover Page 

. Responsibility: Sterling/Tripp 

•Contents: See Illustration C 



Introductory statement 

, Responsibility: Economist... 

* Contents: A brief introductory statement to the 

segment — same as topic introduction 
but at the Segment level. Remember that the 
student may have read the topic introduction 
on a different day or study session; the seg- 
ment introduction may therefore repeat some of 
the idea in the topic introduction. 

Objectives and Instructional Events 
. Responsibility: Economist 

First Objective (bottom of hierarchy) 

(1) Question format 

(2) Narrative 

. recall 

. explain terms, etc. 

. present facts 
. give examples/illustrations 

(3) Practice problems 

(4) References to texts 

Note : 

(1) "Recall 

a. only what has been previously learned 
(reference the segment or objective) 

b. may use "economics terms" which have 
been previously learned; distinguish 
from previous usages 

c. all sub-concepts should be "recalled." 

(2) "Explain terms" 

a. no economic terms -- build bridge from 
everyday English to economic concept; 
no "economese. " 

b. explain what things are, what they are 
*not; what they include and what they 

exclude; give definition or description 
or illustration of all key phrases in 
"performance" part of objective. 

(3) "Present facts" - or answer to question 

in clear simple English 

(4) "Examples and illustrations" 

a. ' give "for instances," work through 

typical problems explaining steps 

b. induce transfer by giving several example 
from different perspective 

c. examples should be clear, concise and 
relevant 

d. interrelate with previous material. 



( 5 ) 



"Practice" . 

a. Give 3-5 practice problems and/or 
questions; "force" s.tudent involve- 
ment and participation 

b. Induce transfer by giving problems with 
different backgrounds , etc. 

c. Immediate knowledge of results through 
Dymedia; number sequentially throughout 
segment. 

d. Motivate and interest students (the 
behavioral objective simply states the 
activity on the part of the learner which 
will be accepted as evidence of his 
having achieved the desired learning; 
it is the instructional events , and in 
particular the "practice/' which must 
ensure that the student really under- 
stands what he is doing when he /[takes 
the required response.) 

(6) "Reference to texts" 

Give references to standard text book 
discussions of the. subject; at least to 
McConnell/ Samuelson. ' 

H. Self Test and Prescription 

. Responsibility: Sterling/Furey - Economists 

. Contents: Segment tests to be provided; 

economists review for content validity 
and recommend changes. 

. Prescriptions: economists recommend "remedial study 

for test items (or groups of test items) 
missed. 

I. Topical Hierarchy 

. Responsibility: Sterling/Tripp 

J. Topic Summary 

. Responsibility: Economist 

.-Contents: one or two page narrative review of all 

segments and objectives in the topic; 
overview; weave the pieces back together. 



Inputs to Economists* 

A. Course Segmentation Syllabus and behavioral objectives 
for other segments (as required) . The economist is 
responsible for ensuring that prerequisite material 
has either been covered in previous segments or is 
presented in the ‘segment which he is preparing. 



B. Segment Topic Introductions ■ 

Preliminary versions will be provided; Economist 
should read the introduction of all related segments, 
resolve outstanding questions he may have, and re - 
vise the introduction to the topics and/or segments 
which he is preparing, as appropriate. 

D. Behavioral Objectives and hierarchy - 

Review, any change must have prior OK of Fitzpatrick; 
use "change procedure." 

E. Criterion Test Items 

1. For reference only (i.e., not to be used for 
practice or example problems) 

2. Economist should review for content validity 
and revise as necessary; revisions of test 
items require approval of M. Furey* — submit 
with recommended changes. 

* 

F* Segment Test 

Preliminary test is provided; economist should 
review for content validity, etc.; changes require 
approval of M. Furey — submit with recommended changes 



VII Concept Summary 



A* Input 

1. Preliminafy introductions 

2* Notes to Economists 

3. Prelim, segment test 

4. B.O.'s and Hierarchy 

5. Criterion test items 

Auxilliarv 

6. Segmentation syllabus 

7. Model of output 

8. Other segment B0‘ s as requested 



poor origin, 
available at 
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B. Output (deliverable item) 

1. Final version of introductions 

2. Recommended changes in BO's and/or hierarchies 

3. Recommended changes to criterion 

test items 

4. Instructional events for each objective 

5. Topic summary 

6. Discussion references and prescriptions 



VIII Operating Procedure 

A. Assign topic to economist 

B. Review intros, objectives and hierarchy 

C. Write Topic Intro 

D. Write Segment Intros 

F. Recommend BO changes 

G. Write Instructional Events 

H. Review test items and recommend: 

1. Segment tests and Prescription 

2. Criterion items review 

I. Write Summaries 

J. Review and format total package 

K. Final review by in-house economists 



XI Discussion of milestones, etc., importance of meeting 
commitments. 



XII Role and Responsibilities of Personnel 

A. Fitzpatrick 

B. Furey 

C. Tripp 



XIII Miscellaneous 

A. Creativity 

. Ideas 

. In the Example/Practice 

B. No Royalties 

Public Domain 
O C. Copyright Problem 

KJC 



Illustration A 



INTRODUCTION TO SPECIALIZATION 



Learning Segments 6/7 
Introduction to Economic Analysis 



Prepared for the 
United States Naval Academy 

by 

Sterling Institute of 
Washington, D.C. 



Illustration B 



S PECIALIZATION 

Contents 



Page 

Cover Page ' . i 

Table of Contents ii 



f eneral Instructions , iii 

ntroduction to Topic 1.0 

Segment X Cover Page 



Introductory Statement 



2.0 



Objectives and Instructional Events 

0515 Description of Absolute Advantage 

0503 Identify Producer with Absolute Advantage 

0508 Description of Comparative Advantage 

0520 Basis of Comparative Advantage 

0506 Identify Producer with Comparative Advantage 

0500 Identify Types of Advantages and Basis for 

advantages in various situations 

Self Test ‘and Prescriptions 

Segment Y Cover Page 

Introductory Statement 

Objectives and Instructional Events 

0510 Definition of Opportunity Cost 

0505 Basis for Specialization and Trade 

0519 Analyze Producers to determine basis for trade 

0521 Results of Specialization under conditions of 

comparative advantage 

0517 Rationale for trade under conditions of 
absolute advantage 
. 0507 Benefits of Specialization 

0509 Drawbacks of Specialization 

0501 Rationale for Specialization 



2 



2 . 



3.0 



3. 



Self Test and Prescription 3. 
Topic Hierarchy Chart . '4. 
Topic Summary 5. 
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SEGMENT 6 

INTRODUCTION TO SPECIALIZATION 
(Part 1 of 2) 



Dymedia Settings : 
Learning - e 
S elf Test - L 
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PINAL CHECK LIST 



Economics Course Materials 



In troduction to Topic 

" A general statement of topic? ”, 

* * 

Relation to other topics? 

___ What student will be able to do when he completes topic? 

+ + + * 

Sequence and/or organization of materials to be presented? 

Possible difficulties? . 



Introduction to Each Segment 
Transition from topic introduction? 

Segment Objectives in General 
^ Every objective directly addressed? 



Individual Objectives 



Prerequisite concepts recalled and restated, if 
covered earlier? 



Prerequisite concepts explained in everyday terms, 
if not covered earlier? 

No new technical, terms used without explanation? 

Examples in sufficient quantity (usually two or more) 
and diverse enough to stimulate generalization? 

Practice problems in sufficient quantity (usually 
three or more) and drawn from diverse backgrounds? 

Practice problems numbered sequentially throughout 
the segment, with first problem in each segment 
numbered “1"? 

Answer choices in practice problems so lettered as to 
fit pymedia Answer pattern? 

References to Samuelson and/or McConnell for each objective 
All test items reviewed and approved or changed? 



Topic Summary 



Checked against topic introduction? 






Guidelines for Economist Writers 



A . General Instruction s 

1. Materials should be written in a clear, narrative style, 
in complete sentences in good English. 

2. Writers should avoid the use of "economic jargon" which 
could confuse students in introductory economics courses. 
The materials, with only a fev; editorial revisions, will 
be given* to the students just as the writers have prepared 
them; therefore, it is imperative that .the content is 
correct and readily understandable. 



3. Handwritten manuscripts are acceptable, providing the 
writing is legible and on only every other line. Type- 
written manuscripts should be double spaced. 



4. Writers must include al^l of the objectives provided them 
in the materials . _ l£the writer is convinced that addi- 
tional information must be included in order to clarify 
or enhance the prescribed objectives, then he should 
include such information. Additions are desirable and 
expected as has been indicated in the "Notes to Econo- 
mists." Should the writer decide that a change in the 
wording of an objective is imperative and that the new 
language is not equivalent to that in the original version, 
then he should contact Mr. Fitzpatrick (338-8800, Ext. 37) 
concerning the proposed change prior to incorporating it 
in the materials. 

5. Test items A, B, C must not be used as illustrations and 
practice problems as they have already been used in segment 
and unit tests, pretests and post tests. 

After the materials have been written, please indicate 
by writing "yes" or "nb" on- each A, B, or C test item 
whether the item. tests the objective to which it is keyed. 
If the writer feels that a distractor is incorrect or that 
the item can be improved in any way, he should write all 
suggestions, including new distractors, etc., below the 
item. 

6. In writing about eighty percent of the materials, writers 
should follow steps 1 through 12 below. The other twenty 
percent will require a deviation in the steps as indicated 
in steps 13 through 17 or in steps 18 through 21. Objec- 
tives requiring a change will be indicated in the segment 



ki t . 



Special attention should be given to providing a rich 
supply of examples , (real-world, relevant) illustrative of 
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the principles or concepts to which the objectives of a 
given topic are addressed, if copyright material is used, 
then permission should be obtained and the appropriate 
references given. 

7. Be sure to complete the chock list and return it with the ■ 
manuscript and the original contents of the kit to Sterling 
Institute . 



Steps in the Preparation of Materials 

1. Check the kit you receive to see that it contains: 

a. ’ Cover page 

b. Table of contents 

c. General instructions 

d. Preliminary introduction to the topic and/or to the 
segment 

e. Hierarchy chart for each segment in the topic 

f. Three test items (A, B, C) for each objective in each 
segment 

g. Segment test for each segment 

h. Notes to Economists 

i. A list of objectives requiring Sample B or C 

2. Read the preliminary introduction, all objectives and test 
items for the topic. This will provide a general overviev; 
of the learnings the student must master and the type of 
question he will be expected to answer when tested. 

3. Road the "Notes to Economists." These notes are not all 
inclusive but point out learnings which should be reviewed 
or recalled, or additional information which should be 
included to facilitate the learning of the objective. It 
is recommended that the writer star or in some way mark 
each objective for which there is a note, as a reminder to 
consider those suggestions when writing. 

4. Begin the narrative with an introductory statement of a 
few sentences which relate the segment to the topic and 
lead in to the first objective (See Sample A, p. ). The 
writer may prefer to write an introduction to the topic 
prior to writing the introduction to the segment; however, 
it seems preferable to write the topic introduction after 
the materials have been completed to avoid the necessity 
of revisions. 

5. Introduce the first objective (abbreviated title) . This 
may be in the form of a question or of a title. Place 
the number of the objectives to the left of whatever means 
is used. (See Sample A, p. .) 
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6. Continue narrative. If it has been indicated t.iat the 
objective should be handled according to* Sample B, then 
follow steps 13 Hi rough 17. If Sample C has been indi- 
cated, follow steps through/'/ . Otherwise, continue 
by recalling information previously learned or by ex- 
plaining new terms. The "Notes to Economists" should 
be of assistance at this point. (See Sample A, p. .) 

7. Present at least two examples or illustrations to clarify 

I the principle to be learned. In some instances, where 

the objective is complex, as many as four, five or six 
examples may be necessary to illustrate, all points. The 
examples should be relevant and interesting to college 
students. If the writer has any unique devices or methods 
which he has found effective in teaching a particular 
objective, he should feel free to include these. 

8. State the definition, principle, etc.,' at an appropriate 
point, and enclose the statement in a- box. Be sure that 
the language used is the same as or equivalent to that 
used in the objective. (See Sample A, p. .} 

9. Present two or more problems which require the application 
of the principle. The number of problems will vary with 
the complexity of the principle, a complex principle re- 
quiring a wide variety of applications. 

10. -Provide answers in the narrativ e to those problems which 

require the student to fill in blanks or place a check in 
front of an answer. Answers should immediately follow the 
problem. (See Sample A, p. . } 

Other problems should be keyed to a Dymedia setting. 

(See Sample A, p. .) 

11. Provide a reference for additional information. (See 
Sample A, p. . ) 

12. Continue narrative with a lead in to the next objective. 

If the objective should be handled according to Sample B, 
follow steps 13 through 17; if according to Sample c, follow 
steps 18 through 21. If no particular plan is recommended, 
repeat steps 5 through 12 until all the objectives have 
been accounted for. 



Begin the narrative with examples of the concept to be 
learned. Then, link the examples to the concept. (See 
Stunple B. p. .) 



13. 



14 
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Underline or enclose in a box the concept to be learned. 



